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Preface 


Hie  leport  advocates  a  promising  appnM^  to  reducing  the  reliability  and 
maintainability  burden  associated  with  advanced  weapon  systems  such  as  die 
Apache  helicopter  and  the  MlAl  tank.  The  approach,  called  "die  maturation 
development  process,"  focuses  on  improving  die  maintainability  of  die  critical, 
high-tech  components  of  a  weapon  system  throughout  its  lifecycle.  The  benefits 
of  die  approach  are  the  adiievement  of  full  designed  performance  and  of  savings 
in  support  costs. 

The  researdi  reported  here  was  conducted  under  a  project  entided  An  Evohring 
Action  Plan  for  Impiementing  V/eapon  System  Management  Concepts  in  Future  Army 
Environments;  however,  die  research  is  highly  syndietic  and  also  draws  upon 
results  from  several  previous  RAND  studies.  The  project  was  joindy  sponsored 
by  die  Strategic  Logistics  Agency;  die  Army  Materiel  Command,  and  the 
Combined  Arms  Su|^x>rt  Command.  The  project  funding  was  supplemented  by 
the  Strategic  Logistics  Agency.  The  project  is  managed  within  die  Military 
Logistics  Program  of  the  Arroyo  Center,  directed  by  Dr.  John  Halliday. 

The  research  should  be  of  interest  dirou^iout  the  Army  logistics  and  acquisition 
commimity. 


The  Arroyo  Center 

The  Arroyo  Center  is  the  U.S.  Army's  federally  funded  researdi  and 
development  center  (FFRDQ  far  studies  and  analysis  operated  by  RAND.  The 
Arroyo  Center  provides  die  Army  with  objective,  independent  analytic  researdi 
on  major  policy  and  organizational  concerns,  enqihasizing  mid-  and  long-term 
problems.  Its  research  is  carried  out  in  four  programs:  Strategy  and  Doctrine, 
Force  Development  and  Technology,  Military  Logistics,  and  Manpower  and 
Training. 

Army  Regulation  5-21  contains  bask  policy  for  the  conduct  of  the  Arroyo  Center. 
The  Army  provides  continuing  guidance  and  oversi^t  through  die  Arroyo 
Center  Policy  Committee  (ACPC),  whidi  is  co<haired  by  die  Vice  Chief  of  Staff 
and  die  Assistant  Secretary  for  Research,  Develqnnent,  and  Acquisition. 

Arroyo  Center  work  is  performed  under  contract  MDA903-91-C-0006. 
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The  Arroyo  Center  is  housed  in  RAND's  Army  Research  Division.  RAND  is  a 
private,  nonprofit  institution  that  conducts  analytic  research  cm  a  wide  nmge  of 
public  policy  matters  affecting  die  naticm's  security  and  welfare. 

James  T.  Quinlivan  is  Vice  President  for  the  Army  Research  E>ivisian  and 
Director  of  die  Arroyo  Center.  Those  interested  in  further  information  about  the 
Arroyo  Center  should  contact  his  office  directly: 


James  T.  Quinlivan 
RAND 

1700  Main  Street 
P.O.  Box  2138 

Santa  Monica,  CA  90407-2138 
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Summary 


Now  is  a  critical  time  for  the  Army  to  understand  and  address  the  especially 
difficult  R&M  (reliability  and  maintainability)  duUenges  presented  by  hig^i>tech 
components.  New  weapon  systems  will  increasingly  depend  on  such 
components  in  order  to  achieve  the  technological  "margin  of  superiority"  that 
U.S.  forces  have  relied  up<ni  to  overcome  quantitatively  superior  foes.  Widiout 
effective  management  of  the  R&M  of  these  systems,  dieir  fuU  designed 
performance  may  not  be  achieved  and  support  costs  may  greatly  exceed 
projected  budgets. 

The  objective  of  fiiis  reseatdx  is  to  develop  policies  and  procedures  to  help  die 
Army  reduce  die  burden  caused  by  die  failures  of  certain  higjh-tech  (chiefly 
digital),  hi^-cost  Class  DC  components.  Taken  togedier,  diese  policies  and 
procedures  compose  an  approach  to  weapon  system  design  and  redesign  diat  we 
term  "maturation  development"  Maturation  development  seeks  to  improve  the 
detection,  reporting,  isolating,  and  removing  of  component  faults;  it  also 
identifies  and  implements  changes  to  component  design  that  improve  R&M. 
Maturation  development  can  be  applied  both  to  new  systems,  such  as  die 
proposed  Comanche  helicopter,  and  to  major  modifications  ("upgrades"  of 
fielded  systems,  such  as  die  Apache  helicc^ter  or  MlAl  tank. 

The  R&M  Challenge  to  Sustaining  High-Tech  Weapon 
Systems 

High'tech  equipment  largely  comprises  digital  components-compact,  complex, 
multifunction  circuit  cards.  Such  components  exhibit  R&M  problems  of  a 
fundamentally  different  and  more  difficult  character  dian  dioae  exhibited  by 
mechanical  equipment  In  mechanical  equipment,  R&M  problems  are  dominated 
by  wear  processes  and  by  physical  failures.  Such  problems  are  usually 
strai^tforward  to  detect  and  isolate,  either  visually  or  widi  simple  test 
equipment  Repair  parts  tend  to  have  low  unit  costs. 

In  high-tech  equipment  by  contrast  many  "failures"  are  best  characterized  as 
instances  of  degraded  performance  that  appear  mteimittendy  and  may  be  rdated 
to  operational  and  environmental  conditims.  Such  faults  are  often  difficult  to 
detect  because  the  cemditions  under  which  dwy  occur  cannot  be  replicated  at  the 
maintenance  facility,  and  dwy  are  often  dif&ult  to  isedate  because  of  dte  complex 
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interrelations  of  circuit  boards.  Moreover,  complex  diagnostic  test  equipment  is 
needed  to  assist  in  the  fault  detection  and  isolation,  and  this  eqviipment  itself  is 
also  high  tech  and  subject  to  R&M  problems.  Replacements  tend  to  be  extremely 
expensive,  making  large  stocks  prohibitively  expensive.  In  addition,  over  time 
specific  components  may  become  "lemoi^s,"  chronically  faulty  units  that  move 
constantly  back  and  forth  between  repair  echelons  and  weapcm  systems, 
delivering  little  of  dieir  full  designed  performance  while  consuming  a 
disproportionate  amount  of  support  resources. 

These  high-tech  R&M  challenges  present  the  double  risk  of  achieving  lower 
weapon  system  availability  at  increased  costs.  Components  that  are  not  faulty 
may  be  mistakerUy  removed  from  weapon  systems,  while  odier  removed 
components  may  be  returned  to  weapon  systems  without  being  properly 
repaired.  Support  costs  rise  because  of  repeated  and  unnecessary  actions  and 
because  of  the  cost  of  additioruil,  very  expensive  spares.  At  the  same  time, 
weapon  system  availability  is  overstated  because  some  of  the  systems  diat  are 
considered  fully  mission  capable  actually  contain  (me  or  more  faulty 
components. 

Maturation  Development 

Because  of  the  avionics  suites  in  its  most  advanced  aircraft,  the  Air  Force  has 
acquired  a  great  deal  of  experience  with  the  R&M  challenges  posed  by  high-tech 
components,  and  RAND  has  worked  with  he  Air  Force  for  over  a  decade  (m 
acquisition  and  support  strategies  for  hi^tech  aviation  electronics.  These 
strategies  fcxrus  on  minimizing  fault  detecticm  and  isolaticm  problems  and 
improving  the  logistics  supportability  of  fielded  systems.  More  recent  RAND 
research  for  the  Army  indicates  that  advanced  Army  systenrs  such  as  he  Apache 
and  MlAl  tank  evidence  R&M  challenges  that  are  fundamentally  he  same  as 
hose  experienced  by  USAF  systems.  The  Army  can  benefit  from  he  Air  Force's 
experience  and  previous  research  investments. 

Based  on  its  research  into  he  high-risk  R&M  characteristics  of  high-tech  weapon 
systems,  RAND  has  developed  a  new  incremental  pr(x»ss-called  "maturaticm 
development"-for  numaging  he  development  and  sustainment  of  R&M  over  he 
life  cycle  of  a  complex  weapon  system.  The  goal  of  maturation  development  is 
he  delivery  of  full  designed  performance  of  he  weapon  system  under  mission 
conditions  and  he  rapid  restoration  of  full  design  performance  when 
malfimctions  (x:cur.  It  provides  for  early  resolution  of  systemwide  R&M 
problems  and  he  identification  of  lemon  components  that  make  heir  way  to  he 
field.  Early  identificaticm  and  resolution  of  R&M  problems  can  be  a  highly 


effective  means  of  understanding  the  cost  drivers  of  readiness  and  sustainment 
and  of  managing  the  support  cost  of  the  weapon  S3retem.  Maturation 
development  can  be  applied  to  both  new  systems  and  to  fielded  systems. 

For  new  weapon  systems,  maturation  development  begins  early  in  the  life  cycle 
during  research  and  development.  Ccmtinuous  data  collecbon  on  the 
performance  of  components— even  prototypes— during  this  period  provides  the 
basis  for  a  description  of  the  physics  of  failure  modes  that  is  important  for  later 
ar\alysis.  These  early  data  are  also  important  input  fcr  the  estimation  of  R&M 
budgets. 

Maturation  development  of  new  systems  intensifies  during  low-rate  production 
and  early  fielding.  The  approach  calls  for  a  dedicated  period  of  intense 
operation,  data  collection,  and  aruilysis  immediately  upon  fielding  a  weapon 
system.  This  period  would  occur  during  the  latter  portion  of  the  engineering  and 
manufacturing  development  phase,  now  called  die  low  rate  of  initial  production 
(LRIP)  period.  The  purpose  of  this  period  would  be  to  detect  and  isolate  design 
(ieficiepcjpfi  by  intensively  operating  the  components  in  a  fixed  configuration 
within  the  environment  where  they  will  normally  operate.  Ald\ough  the 
maturation  development  period  coincides  with  operational  test  and  evaluation,  it 
would  be  primarily  development  testing  conducted  (observed)  and  supported  by 
engineering  personnel,  followed  by  a  detailed  analysis  of  the  data  gathered.  The 
output  of  the  maturation  data  collection  period  is  a  set  of  engineering  changes  to 
modify  the  full-rate  production  version  of  the  components  so  that  the  major 
R&M  problems  are  resolved. 

Maturation  development  continues  into  hi^rate  production  and  fielding.  The 
database  set  up  in  the  early  acquisition  phases  and  used  extensively  in  the 
maturation  development  period  would  then  be  used  to  monitor  performance  and 
to  identify  and  isolate  lemon  components.  This  continuing  effort  permits 
analysis  of  the  effects  of  configuration  changes,  of  aging,  and  of  new  mission 
requirements  or  operational  environments. 

An  alternative  implementing  strategy  for  maturation  development  applies  to 
systems  that  are  already  fielded— and  for  which  only  a  limited  database  exists. 
These  hi^-tech  systems  are  likely  to  still  exhibit  less  maturity  ttian  one  would 
desire  in  a  high-tech  weapon  system.  Indications  of  an  immature  system  include 
a  high  degree  of  difficulty  in  favdt  detection  and  fault  isolation,  along  with  a  set 
of  components  that  consume  large  amounts  of  logistics  resources.  This  strategy 
rails  for  the  establishment  of  a  limited  database  to  identify  die  extent  of  the 
problem  and  to  isolate  the  most  immature  electronic  units.  A  performance- 
oriented  tracking  of  the  equipment  repair  system  would  allow  die  weapon 
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system  manager  to  identify  which  components  are  consuming  die  most  logistics 
resources  and  focus  his  attention  on  those  components. 


Information  Requirements  to  Support  Maturation 
Development 

A  key  element  of  the  maturation  development  process  is  a  well-developed 
management  mformation  system  Uttked  to  an  integrated  R&M  database  to 
facilitate  efficient  and  effective  resolution  of  the  R&M  problems  associated  with 
high-tech  components.  The  database  should  be  integrated  across  functions  and 
echelons  as  well  as  through  time.  An  integrated  system  is  needed  because,  as 
suggested  above,  R&M  effects  in  a  complex  weapon  system  can  appear  in  the 
system  at  locations  and  echelons  that  are  remote  from  the  carise.  For  example, 
what  appears  from  the  operator's  perspective  to  be  a  radar  problem  may  in  fact 
be  a  power  supply  problem.  Because  fault  indications  in  d\ese  complex  systems 
are  not  always  unambiguous,  patterns  must  be  observed  and  recorded.  This 
complexity  requires  linkages  across  the  elements  of  the  weapon  system  and 
across  time.  The  matuigemen  irdormation  system  must  allow  the  mairager  to 
assess  the  performance  of  elements  of  the  weapon  system  and  support  system, 
which  requires  the  ability  to  link  data  from  operation;  fault  indication  at  the  end 
item,  including  those  provided  by  the  built-in-test  (BIT),  intermediate  test,  and 
repair  and  depot  test;  and  repair  over  a  period  of  time. 

Among  the  data  elements  needed  to  support  maturation  development  are  those 
associated  with  unit-level  and  intermediate  and  depot  or  contractor  maintenance, 
including  data  associated  with  diagnostic  test  equipment  at  all  levels,  with 
administrative  and  transportation  processes,  and  with  maintainers.  For  at  least 
some  selected  component  types,  individual  tracking  by  serial  number  will  be 
required  to  permit  the  identification  and  correction  of  lemorts  as  they  emerge. 

The  maturation  development  process  poses  more  demanding  data  requirements 
than  can  be  met  widi  current  data  systems.  However,  it  is  well  within  tiie  state  of 
the  art  to  incorporate  "smart  chips"  into  the  design  of  high-tech  weapon  systems 
to  lessen  the  data  collection  burden,  perhaps  to  the  point  that  it  is  even  less  of  a 
biuden  than  currently  exists. 


Conclusion 

We  expect  the  need  for  maturation  development  to  increase  in  the  1990s.  The 
Army  already  needs  to  achieve  required  weapon  system  availabilities  at  lower 
costs.  Moreover,  tire  Army  can  expect  to  encounter  increasing  R&M  problems  as 
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its  inventory  of  weapon  systems,  both  through  upgrades  and  new  procurements, 
becomes  increasingly  sophisticated  technologically. 

In  implementing  maturation  development,  the  Army  faces  four  lands  of 
obstacles,  each  of  which  can  be  overcome: 

•  Cost  obstacles:  Maturation  development  requires  some  increase  in 
development  costs.  Since  these  costs  must  be  wei^wd  against  savings  in 
support  costs  over  the  system's  life  cycle,  it  is  important  to  establish  the 
capability  to  measure  support  costs  accurately.  The  operatiorud  benefits  of 
improved  high-tech  R&M  performance  must  also  be  considered. 

•  Techtucal  obstacles:  Maturation  development  requires  the  development  of 
new  measvires  of  R&M  perfomumce  and  improved  data  coUecticm  and 
analysis  capabilities.  Irmovatioirs  in  automated  data  collection,  including  die 
use  of  "smart  chips,"  may  greatly  reduce  the  data  collection  burden. 

•  Envirorunental  obstacles:  Maturation  development  can  be  expected  to  delay 
the  fielding  of  a  new  system  or  upgrade  for  a  few  years.  During  the  Cold 
War,  procurement  processes  were  hi^y  compressed  because  of  competition 
from  the  Soviet  Union.  Widi  toda)r's  reduced  risk,  there  should  be  less 
resistance  to  maturing  systems  before  fielding  diem. 

•  Organizational  obstacles:  Maturation  development  requires  a  closer 
integration  of  engineering  and  logistics  activities  and  of  organic  and 
contractor  activities.  Incentive  systems,  including  contracts,  may  need  to  be 
changed  to  reflect  an  increased  emphasis  on  improving  the  feiult  removal 
efficiency  of  die  maintenance  system  and  die  availability  of  fully  missicm 
capable  weapon  systems  while  reducing  total  life  cycle  costs. 

The  potential  benefits  of  implementir^  die  maturation  development  process  for 
managing  the  R&M  of  higjh-tech  weapon  systems  are  twofold:  first,  die 
achievement  of  full  designed  system  performance,  which  contributes  to  genuine 
mission  capability;  and  second,  reduction  in  support  costs  over  the  life  cycle  of 
the  system. 
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Symbol 

-A 

-AF 

Bad  Actor 

BIT 

BITE 

CAMS 

Class  DC 

CONUS 

DoD 

DRIVE 

Dyna-METRIC 

ECM 

ECP 

EETF 

FMC 

FRA 
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LPRF 

LRIP 

LRU 
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MEP 
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MTBF 

MTBR 

N- 

NEOF 

NRTS 

ODS/S 

R 

R&D 

RAM 

R- 

RBM 

REMIS 

SARDA 

SRU 


Definition 

Aimy*sponsored  leseatdt 
Air  Foice-sponsored  researdt 
See  Lemon 
Built-in-test  (system) 

Built-in-test-equipment 
Core  Automat^  Maintenance  System 
Reparable  weapon  system  coaq>onents 
Continental  United  States 
Department  of  Defense 

Didiibutkm  and  repair  in  variable  environments 
Dynamic  Multi-Echekm  Tedtnique  for  Recoverable  Item 
Control 

Electronic  countermeasure 

Engineering  Change  Prt^xjsal 

Electronic  equipment  test  fodlity 

Fully  mission  capable 

Forward  repair  activity 

Heads-up  display 

Integrate  logistics  support 

Chronically  faulty  component 

Light  heliorpter  erq^erimental 

Low-power  radio  frequency 

Low  rate  of  initiai  production 

Line  replaceable  unit 

Laser  transceiver  unit 

Maintaiiudrility 

Mission  Equqinanent  Package 

Mul%le  Laundt  Rocket  System 

Mean  time  between  fiUhire 

Mean  time  behveen  removal 

RAND  Note 

No  evidence  of  failure 

Not  reparable  ttiis  station 

Operation  Desert  Shield/Operation  Desert  Sttmn 

Reliability 

Research  and  development 
Reliability  and  onaintairudrility 
RAND  Report 

Readiness-based  onainterumce 
Reliability  and  Maintainability  bifonnaticm  System 
Secretary  of  the  Army  for  Research,  Development  and 
Acquisition 
Shop  replaceable  unit 


XX 


TADS/PNVS 

Target  Acquisition  and  Designation  Si^t/Pilot  Nig^t 
Vision  Sensor 

TBO 

Time  betweoi  overhaul 

TEU 

TADS  electrocucs  unit 

TICARRS 

Tactical  Interim  CAMS  and  REMIS  R^>orting  System 

TMDE 

Test,  measurement,  ai\d  diagnostic  equipment 

Type  A 

A  fault  (hat  is  definite  and  repeatable 

TypeB 

A  fault  dut  is  intermittent  and  difficult  to  isolate 

USAF 

United  States  Air  Force 

WSSM 

Weapcm  System  Sustainment  Management 

1.  Introduction 


Now  is  a  critical  time  for  the  Army  to  understand  and  address  die  c^MciaUy 
difficult  leliaMlity  and  maintainability  (MeM)  challenges  presented  by  high-tech 
weapon  systems.  As  die  Army  moves  fcmvazd  in  die  devdopment  of  new 
weapons  and  die  modification  of  existing  ones,  it  incorpocates  more  hig^ 
technology  in  its  designs.  High>tedi  items  are  paiticulariy  in^portant  in  two 
senses:  First,  they  are  critical  to  adiieving  the  “margin  of  siyctiofity*'  diat 
advanced  tedinology  provides  to  U.S  %veapon  systems;  and  second,  they  are 
extremely  expensive  to  develop,  repair,  and  replace,  lluough  the  next  decade 
and  into  die  next  century,  hig^tech  wcqxxi  systems  wiU  greatfy  doraiiuite  the 
Army  force  structure.  The  Army  will  come  to  depend  on  fsr  fewer  numbers  of 
these  systems  because  of  dieir  lediality  and  will  depend  on  a  mudi  leaner 
logistics  structure,  in  part  because  dieir  fnesumed  reliability  and  ease  of 
maintenance.  The  upgrading  of  existing  systems  and  die  development  of  new 
weapon  systems  require  an  approach  that  provides  die  margin  of  superiority 
expected,  die  availability  of  fully  missian  capable  weapon  systems  needed,  and 
the  ease  of  maintenance  needed  in  a  power  projection  environment 


Purpose 

This  report  advocates  a  new  approach  to  enhancing  die  RfcM  performance  of 
high-tech  weapon  system  compcaients  (te.,  Qass  DC  conqxments).  Analyses  of 
die  proposed  Comanche  helicopter  suggest  diat  eight  high'tedi  components  will 
cause  70  percent  of  the  downtime.^  Fielded  high-tedi  weapon  systems  sudi  as 
the  Apache,  MlAl,  F-15,  and  F-16  have  similar  dominating  conqxments  with 
such  features.^  The  new  approach  to  managing  RScM,  whidi  we  term 
"maturation  devdc^nnent,"  has  die  potential  to  increase  weapon  system 
availability  (and  hence  operational  capability)  idiile  also  reducing  total  life  cycle 
costs.  The  approach  can  be  applied  both  to  new  weqxai  system  acquisitions  and 
to  major  modifications  (upgrades)  of  fielded  we^ion  systems. 


^ForrdaicdanalyMs,MeB«niianctaL(19S^.  Iteattaamcpocied  here  docs  not  appcv  in 
ttatrepoit.  It  wuunooverad  in  fataranalyiahjr  the  project  taam. 

^Robbins  ct  aL  (1991);  Bennan  et  aL  (1988);  Geboun  ct  aL  (1989). 
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Increased  Weapon  System  Availability 

Historically,  the  current  approach  to  acquisition  numagement  has  t)q>ically 
achieved  a  great  deal  during  the  preproduction  phases  of  the  system  life  cycle  in 
terms  of  improving  weapon  system  availability.  This  has  been  accomplished 
through  a  strong  emphasis  on  improving  the  reliability  of  high*tech  ccnnponents. 
However,  the  benefits  that  accrue  from  ttus  approadi  are  often  e)diausted  by  the 
time  of  fielding,  and  the  approach  has  typically  not  been  able  to  adiieve  the  full 
designed  weapon  system  availability  goals  before  the  system  is  fielded. 

Under  maturation  development,  both  reliability  and  maintainability 
improvements  are  sought  during  file  earlier  life  cycle  phases,  but  during  low  rate 
of  initial  production  as  mainteiumce  occurs  and  data  become  available,  file  focus 
shifts  more  acutely  to  maintainability  improvements  before  file  system  is  fully 
fielded.  More  specifically,  maturation  develt^nnent  seeks  to  improve  the 
processes  of  detecting,  reporting,  isolating,  and  removing  faults  so  fiiat  file 
incidence  of  repeated  faults  is  greatly  reduced.  It  also  seeks  to  identify  and 
implement  modificaticHis  to  component  designs  that  exhibit  poor  maintainability. 
The  added  emphasis  cm  maintainability,  we  argue,  will  permit  file  adiievement 
of  full  designed  weapcm  system  availability  and  performance  goals  as  well  as  file 
continuous  improvement  of  maintainability  and  weapon  system  availability  rates 
throu^out  file  life  cycle. 

In  the  case  of  new  weapon  systems,  maturaticm  development  has  major 
implicaticms  for  file  period  of  low  rate  of  initial  production,  where  many  changes 
to  the  current  approach  are  required  to  ensure  fiiat  full  designed  availability 
goals  ate  met  before  high-rate  ptoducticm  and  fielding  and  fiiat  file  support 
system  is  positioned  to  ccmtinuously  inqirove  weapon  system  availability  during 
fielding.  In  addition,  fiiis  approadi  has  implications  for  file  preproducticm 
phases  of  file  life  cycle,  where  some  steps  (e.g.,  data  coUecticm  and  integration) 
must  be  taken  in  preparaticm  for  fiw  later  focus  cm  improving  maintainability. 

In  the  case  of  major  upgrades  to  fielded  weqmn  systems,  maturaticm 
development  has  major  implicaticms  for  modification  implementaticm.  As  with 
new  weapcm  systems,  during  fiw  period  preceding  full-rate  production  and 
retrofitting  fiiere  are  significant  changes  to  fiw  accjuisition  strategy.  Because 
mwy  of  file  upgrades  will  include  ciiai^;es  to  fiw  high*tech  electrcmic  suites,  fiw 
strategy  should  include  planning  for  a  maturation  of  fiw  design  oi  these 
components  and  the  use  of  test  beds  to  fully  understand  the  capabilities  and 
effects  of  fiw  prc^iosed  design. 
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Reduced  Total  Life-C^le  Costs 

Maturation  development  is  also  expected  to  reduce  total  life-cycle  coats  under 
most  circumstances.  For  example,  wlwn  this  approach  is  applied  to  a  new 
weapon  system  or  major  upgrade,  more  costs  will  accrue  before  full  producticm, 
because  of  foe  increased  activity  focused  on  improving  maintainability;  however, 
during  foe  fielded  life  of  the  new  or  modified  system,  costs  would  be  mud\ 
lower,  reflecting  the  maintainability  improvements.  Fewer  maintenance  actions 
mean  fewer  test  stands  and,  most  important,  fewer  maintainers. 

When  maturation  development  is  applied  to  a  fielded  system,  there  is  less 
potential  for  cost  savings  over  the  remaining  life  cycle.  Careful  case-by-case 
aiudysis  would  be  required  to  determine  whether  the  benefits  of  maturation 
development,  bofo  in  terms  of  cost  and  performance  gains,  outweigh  foe  costs. 

In  foe  Service's  limited  experience  in  applymg  elements  of  the  nuituration 
approach,  there  have  been  instances  in  whidi  some  dumges  (redesign  of  specific 
comprments)  to  improve  R&M  performance  were  identified  but  judged  too 
expensive  to  implement.  This  judgment  was  based  primarily  on  consideration  of 
the  size  of  foe  fleet  that  required  retrofitting. 

It  is  important  to  note  foat  foe  expected  cost  savings  would  not  be  fully  captured 
using  foe  current  data  collection  systems.  CXir  researdi  indicates  foat  a  large 
|X)rtion  of  the  mainteiumce  costs  associated  wifo  hig^-tedi  components  are  due 
to  actions  taken  in  response  to  degraded  ^tem  performance,  such  as  multiple 
removals  and  tests  in  search  of  an  elusive  fault,  actions  that  are  not  currently 
captured  for  diagnostic  use. 


Should  Weapon  Systems  Need  to  Be  Matured? 

An  obvious  objection  to  foe  maturation  develqmnent  process  is  foat  it  should  not 
be  necessary  if  a  weapon  system  is  designed  aiui  built  properly  in  foe  first  place. 
Therefore,  according  to  this  argument,  improvements  should  be  sou^t  in  those 
activities  first. 

This  objection  is  based  on  an  incomplete  understanding  of  foe  complex  ruiture  of 
foe  performance  degradations  in  high-tech  weapon  system  cooqxments.  It  is 
important  to  recognize  foat  foe  hig^iest  quality  engineering  and  manufacturing 
carmot  obviate  foe  rteed  for  a  maturation  period  after  foe  research  and 
development  phases.^  As  is  demonstrated  in  Section  2,  a  simple  cause  and  effect 


^Thmaie  a  nuinber  of  analogues  to  nutimtionclevelopawnt  in  commercial  indumies.  Oneis 
foe  practice  of  softwara  finns  in  dinributing  "beta”  venkma  of  new  pnxhids  to  a  limited  set  of  users. 
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relationship  between  removals  of  components  and  a  hard  failure  does  not  exist  in 
die  comptex  interdependency  of  an  integrated  electronic  system;  rather, 
anomalies  in  hardware,  software,  and  environment,  and  die  effects  of  operational 
missions  interact  to  produce  performance  degradations.  This  ccnnplexity— and 
the  complexity  of  the  built-in-test  (BIT)  S3rstem— results  currendy  in  removal 
processes  that  are  frequenUy  not  linked  to  the  underl3ring  physical  causes  of  the 
lack  of  performance  in  the  weapon  system.  Furdier,  these  degradatians  are 
frequendy  not  reproducible  in  die  relativdy  quiescent  maintenance  environment. 
The  diagnostic  equipment  used  at  hig^r  echelons  of  maintenance  do  not 
simulate  die  mission  environment  Moreover,  the  usual  assumption  is  that 
higher-level  tests  are  complete  and  diere  is  no  need  to  verify  diat  the 
malfunctions  identified  by  the  higjher-level  test  are  indeed  the  same  pn^lems 
that  resulted  in  the  removals.  This  assumpticm  creates  the  potential  for 
significant  removal  problems  to  go  undetected  in  die  maintenance  tests. 

Maturation  development  relies  strongly  on  a  period  of  intensive  field  operation, 
data  collection,  and  analysis.  Widiout  a  detailed  field  data  collection  and 
engineering  evaluation  resulting  in  appropriate  hardware  and  software 
modifications,  experience  shows  diat  performance  degradations  are  certain  to  be 
present  in  complex  weapon  systems.  These  anomalies  lead  to  poor  reliability, 
maintainability,  and  availability,  generating  hig^  support  costs  while  reducing 
operational  capability.  Unless  they  ate  removed,  diey  will  persist  for  die  life  of 
the  weapon  system. 


Organization  of  This  Report 

The  remainder  of  this  report  describes  die  maturation  development  process, 
drawing  upon  research  of  bodi  Air  Force  and  Army  weapon  systems.  Section  2 
demonstrates  (1)  diat  hig^-tech  components  have  fiulure  characteristics  that 
differ  fundamentally  from  those  of  mechanical  components,  affecting  operational 
performance  and  complicating  diagnosis  and  repair,  and  (2)  diat  traditional 
approaches  to  managing  R&M,  which  were  developed  for  mechanical  systems, 
are  not  well  suited  to  high-tech  components.  Section  3  describes  maturation 
development  and  expUuns  how  its  approach  to  managing  high-tech  R&M  would 
differ  from  what  is  currendy  done  at  each  life-cycle  stage.  Special  attention  is 


These  users  test  the  new  fwoduct  in  wrhat  could  be  described  as  its 'notmal,  opeiatianal 
environment' and  uncover  bugs  and  design  deficienciet  that  die  fhin  cut  choose  to  conect  befbie 
distributing  the  fuH-market  version.  Anodiet  analogue  is  the  use  o(  pilot  plants  taidw  processing 
industries.  Rather  than  build  a  full-scale  betory  employing  a  new  process,  a  firm  builds  a  small  but 
fully  functional  plant  to  test  the  new  concept  operation^.  What  dwse  practices  have  in  ooinmon  is 
the  recognition  that  complex  mechanisms  cannoi  be  fully  matured  until  infonni^on  is  available 
regarding  how  they  perform  iriiile  operating  normally  in  a  genuine  emrironment 
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devoted  to  the  discussion  of  the  period  of  low-rate  production,  because  ttus  is 
where  the  differences  are  most  pronounced.  Sectum  4  e}q>lain8  die  kinds  of  data 
that  are  requited  to  support  die  new  approach,  arguing  die  need  for  an  R&M 
data  system  that  is  integrated  across  edielons  and  functians  and  over  time. 
Section  5  offers  concluding  observations  and  suggests  directions  for 
implementing  die  maturati<m  development  approach  in  die  Army. 

Appendix  A  sets  the  maturation  development  approach  in  the  context  of  a 
comprehensive  concept  of  logistics  management  under  development  at  RAND: 
Weapon  System  Sustainment  Management  (WSSM). 
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2.  The  R&M  Challenge  to  Sustaining 
High-Tech  Weapon  Systems 


R&M  Problems  Characteristic  of  Mechanical  Equipment 

R&M  problems  faced  by  ttie  Army  reflect  the  technological  sophistication  of 
weapon  systems.  An  army  consisting  solely  of  mechanical  equipment  has  simple 
R&M  problems  dominated  by  wear  processes  and  easily  diagnosed  physical 
failxires.  Mechanical  equipment  is  diaracterized  by  faults  fliat  exhibit  symptoms 
fluit  are  observable  in  ail  operations  of  die  faulty  piece  of  equipmovt,  wheflier  in 
actual  field  use  or  in  test  These  can  be  referred  to  as -Type  A"  foults.  Theefldrt 
of  the  fault  is  usually  seen  close  to  the  locati<m  of  the  failed  part,  and  die  cause 
and  effect  relationship  is  traceable  eidier  visually  or  using  fairly  simple  test 
equipment  Repair  parts  for  mediankal  equipment  tend  to  have  low  unit  costs. 
Moreover,  when  managing  the  R&M  perfomumce  of  mechanical  equipment,  the 
problems  in  different  functional  areas-such  as  testing,  repair,  and  supply~do  not 
usually  interact  closely.  Rather,  a  problem  can  typically  be  identified  cleanly  as  a 
problem  in  a  specific  functional  area  at  a  specific  echelon. 


R&M  Problems  in  High-Tech  Systems 

The  latest  generation  of  weapon  systems  relies  an  hig^  technology-complex, 
integrated,  electronic  subsystems-to  enharKe  mission  performance  and  provide 
combat  multiplier  effects.  Hig^>tech  components  require  sophisticated  built-in 
test  equipment  (BITE)  and  test,  measurement,  and  diagnostic  equqiment  (TMDE) 
to  help  detect  and  isolate  faults  for  repair.  Neverdieless,  die  prevailing 
assumption,  derived  from  experience  with  Type  A  faults,  is  diat  only  one  failure 
occurs  at  a  time  and  that  tests  are  complete  and  accurate.  Driven  by  diis 
assumption,  die  maintenance  of  hig^tech  systems  is  supposed  to  proceed  in  die 
following  fiishicm  (the  current  process  is  represented  schematically  in  Figure  2.1). 

Components  fail  in  operation  and  the  failure  is  detected,  eidier  by  some  BITE  or 
by  die  operator.  The  fault  is  then  reported  by  die  operator  to  a  maintenance 
technician.  The  maintenance  technician  dien  diagnoses  the  feuh  using  some 
combination  of  skill,  technical  data,  and  TMDE.  He  will  dien  either  repair  die 


Figan  2.1 — Schematic  Rcpictentation  of  the  Maintenance  System  for 
High-Tech  Components 


faulty  component  or  replace  it  wittt  a  "good”  one  and  send  the  hiulty  one  to  dte 
next  hi^er  echelon  for  repair. 

At  this  stage,  the  faulty  component  (a  line  replaceable  unit,  or  LRU)  will  also  be 
subjected  to  additional  fault  diagnosis  by  a  maintenance  technician  using  some 
combination  of  skill,  technical  data,  and  TMDE.  He  will  foen  eifoer  repair  dw 
foulty  LRU  by  repairing  a  subcompcHtent  or  replacing  the  subcomponent  with  a 
"good"  one  and  sending  the  faulty  subcomponent  to  die  next  edielon  for  repair. 
The  "good”  LRU  is  dien  made  available  for  subsequent  use  in  repair  of  a  weapon 
system.  The  faulty  subcomponent,  called  a  shcq}  replaceable  unit  (SRU),  is  sent 
for  repair  by  a  maintenance  technician  who  will  diagnose  die  hiult  using  some 
combination  of  skill,  technical  data,  and  foult  detection  and  isolation  equipment 
He  will  identify  a  faulty  part  and  replace  it  widi  a  "good”  part 

Ideally,  every  detected  fault  will  be  reported,  and  every  reported  fault  will  be 
repaired  following  expedient  fault  diagnosis.  Unfortunately,  diat  is  not  the  case 
with  the  maintenance  of  hi^-tech  compcments.  The  foult  removal  process  of 
high-tech  components  runs  into  difficulties  in  all  diree  phases-fault  detection, 
fault  reporting,  and  fiiult  isolation  and  repair.  As  a  result,  die  current  system  is 
flooded  widi  bodi  false  n^tives-cmnponents  diat  are  diought  to  be  broken  but 
are  not-and  false  positives-components  diat  are  diou^t  to  be  fixed  or  "good" 
but  are  not 


8 


These  difficulties  arise  because  high-tech  equipment  is  characterized  by  what  can 
be  termed  "Type  B"  faults:  complex,  mission  and  owironmentally  dependent 
faults  that  do  not  have  statioiuuy  observability.  These  presort  serious  challenges 
to  providing  sustained  weapon  system  availability  and  to  ccmtroUing  support 
costs.  These  intermittent  faults  are  not  systematically  recorded  or  measured  by 
traditional  R&M  testing  because  the  detecticm  systems  are  not  designed  to 
capture  ffieir  occurrence.  As  a  result,  some  hi^-tech  weapon  systems 
considered  to  be  fully  missicm  capable  (FMC)  may  in  fact  contain  components 
exhibiting  intermittent  faults.  Weapon  system  availability  within  a  unit  may  be 
overstated  and  may  not  be  sufficient  to  meet  a  conunander's  operatioiud 
objectives.^ 

LRUs  in  high-tech  electroiuc  subsystems  typically  have  ixtany  SRUs,  most  of 
which  are  multifunction  circuit  cards.  These  circuit  cards  are  a  complex 
integration  of  subcomponents  with  a  high  degree  of  engineering  sophistication 
that  makes  possible  the  state-of-ffie-art  performance.  Usually  each  of  these 
drcxiit  cards  is  designed  to  be  diagnosed  and  replaced  at  the  shop,  if  necessary. 
The  complexity  of  dvese  hi^-tech  SRUs  leads  to  a  diagnostic  problem  where  the 
existence  of  a  problem  is  frequently  disguised  and  the  cause  of  a  malfunction  is 
not  apparent. 

Moreover,  the  range  of  failure  modes  in  high-tech  components  is  very  large, 
especially  as  encoimtered  in  the  field,  and  includes  degradaticnt  modes  that  are 
difficult  to  observe  under  quiescent  conditions.  BITE,  in  particular,  has  limited 
coverage  of  the  performance  spedficatitms  and  limited  capability  to  observe 
performance  in  the  mission  environment 

The  complexity  of  the  BITE  and  TMDE  causes  an  additional  set  of  R&M 
problems:  The  high-tech  components  of  ffie  test  equipment  itself  fail  frequently 
and  caimot  be  repaired  easily.  They  can  be  very  expensive:  For  example,  an 
electronic  equipment  test  facility  (EETF)  that  tests  78  different  types  of  LRUs 
costs  $10  million.  TMDE  has  many  possible  faulty  components,  aitd  most  test 
stands  remain  not  mission  capable  for  long  periods.  Because  so  few  test  stands 
are  available,  bottlenecks  occur  that  cause  many  weapon  system  LRUs  to  be 
placed  on  "awaiting  maintertaiKe"  status  or  sent  to  a  dep>ot  for  repair.^ 


^High*tech  LRUs  tend  to  be  v«y  costly  ($20,000  to  $200,000  each),  which  makes  large  stock  buys 
prohibitively  expensive  as  an  alternative  to  expeditious  repair. 

^  An  analysis  of  the  benefits  of  consolidating  test  equipment  at  higher  echelons  is  presented  in 
Bemtan  et  al.  (1968). 
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In  sum,  high-tech  equipment  exhibits  R&M  problems  of  a  fundamentally 
different  nature  than  those  displayed  by  mechaxucal  equipment.^  Rather  than 
simply  "breaking,"  as  mechanical  components  do,  high-tech  equipment  more 
typically  fails  in  the  sense  that  it  intermittently  exhibits  degraded  or  aberrant 
performance.  These  intermittent  faults  result  in  problems  in  fault  detection, 
reporting,  and  isolatioiL  The  following  subsections  describe  each  of  these 
problem  areas  in  turn. 

Fault  Detection 

Fault  detection  occurs  when  either  the  BIT  detects  an  anomaly  and  the  operator 
observes  the  BIT,  or  when  die  operator  observes  die  anomaly  direcdy.  Several 
factors  contribute  to  the  detection  problem  (including  some  of  the  same  factors 
that  contribute  to  reporting  problems). 

General  Degradation  vs.  Failure.  High-tech  systems  experience  "graceful" 
degradation  by  design,  but  the  degree  of  degradation  is  often  difficult  to 
measure.  In  fact,  some  elements  of  die  sjrstem  may  be  operating  correcdy,  while 
other  elements  of  the  system  may  have  degraded  operation  but  remain 
undetected  by  the  BIT  because  of  varying  degrees  of  degradation. 

Limited  Opportunities.  Restricted  use  of  certain  subsystems-espedally  during 
peacetime  operations-limits  the  opportunities  of  the  BIT  to  detect  faults. 
Different  radars,  for  example,  are  rarely  tested  over  the  full  range  of  design 
performance  duting  peacetime. 

Intermittently  Observable  Symptoms.  Because  many  high-tech  components  are 
sensitive  to  dynamic  environmental  conditicms,  faults  present  in  one  operational 
environment  may  not  appear  in  less  hostile  environments,  such  as  those  foimd  in 
shops  or  in  benign  field  tests.  Some  symptoms  may  be  triggered  by  vibration  or 
temperature  occurring  in  operation  and  cannot  be  duplicated  easily  at  a  repair 
facility. 

Incomplete  and  Impcrf^  BIT  System.  The  BIT  was  developed  to  alleviate 
some  of  die  fault  detection  pn^lems.  It  is,  however,  limited  to  detecting  only 
what  it  was  designed  to  detect.  Sometimes,  it  does  not  detect  a  fault  because  it 
w<is  not  designed  to  test  for  that  fault  At  otiier  times,  it  reports  a  fault  that  does 
not  exist  because  the  BIT  itself  is  faulty.  Many  of  the  BIT  systems  are  not 
comprehensive,  many  of  tiiem  are  not  cmitinuous,  many  do  not  indicate  multiple 


^Gebman  (1989). 
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faults,  and  most  do  not  indicate  the  severity  of  the  fault.  Finally,  because  some  of 
the  faults  are  intermittent,  the  BIT  can  appear  to  be  irregular. 

Fault  Reporting 

Unless  a  fault  is  reported,  weapon  systems  are  considered  FMC  Inaccurate  foult 
reporting  causes  a  skewed  and  potentially  dangerous  view  of  weapon  system 
availability  because  weapon  systems  with  no  reported  faults  are  assumed  to  be 
ready  for  d\eir  mission.  Operators  are  die  principal  source  for  reporting  faults  to 
maintenance.  When  operators  observe  anomalies  and  do  not  report  diem,  no 
maintenance  occurs.  When  operators  do  report  an  anomaly,  but  maintainers  are 
unable  to  cmifirm  it,  the  suspected  component  is  considered  to  be  good. 
Frequently,  die  record  of  die  unconfirmed  anmnaly  is  unavailable  for  future  use 
by  maintainers. 

A  request  for  maintenance  is  influenced  by  many  considerations,  among  diem 
the  following. 

Mission  Requirements.  The  operator  might  dioose  not  to  initiate  a  repair 
request  for  a  system  not  required  for  the  next  mission,  such  as  a  night  vision 
system  for  a  da)rtime  mission.  He  also  may  delay  a  repair  request  if  he  lacks 
confidence  in  the  ability  of  die  maintenance  tedinidaiis  to  repair  die  system,  for 
example. 

General  Degradation  vs.  Failure.  High-tech  systems  rarely  fail  catastrophically, 
but  still  fail— where  failure  means  die  inability  to  provide  the  full  designed 
capability.  Generally,  diese  systems  fall  victim  to  faults  that  degrade  their 
perfomuuice.  More  often  dian  not,  operators  must  make  a  judgment  about  a 
symptom  of  a  fault  (ie.,  degraded  or  inconsistent  performance)  and  determine 
whedier  it  is  serious  enough  to  be  a  reportable  discrepancy.  Thou^  "graceful" 
degradation  is  generally  a  design  criterion,  die  degree  of  actual  degradation  is 
often  difficult  to  judge. 

Limited  Opportunities.  Restricted  use  of  certain  subsystems  during  peacetime 
limiL>  die  operator's  experience  in  determining  a  fault.  Target  acquisition  and 
fire  control  radars,  for  example,  are  rarely  tested  over  the  full  range  of  design 
performance  during  peacetime. 

Operator  Workload.  The  operator  may  be  unable  to  record  (even  verbally)  die 
environmental  conditions  diat  triggered  the  fault  indication  because  it  occurred 
at  a  time  when  there  were  other  demands  for  his  attention.  It  is  easy  to 


undostand  wdty  an  (^leiator  may  be  unable  to  provide  all  die  infonnatkm 
needed  by  a  maintenanoe  tedmidan. 

Fault  Isolation 

After  a  foult  is  detected  and  reported  in  a  hig^tedt  weapon  system,  it  may  be 
difficult  to  isolate  it  to  a  particular  cmnponent  or  set  of  components.  Frequendy 
the  cause  of  the  malfunction  is  ambiguous.  A  complex  electronics  fault  may 
manifKt  itself  in  any  ntimber  of  components  duit  are  linked,  making  isolation  to 
die  malfunctioning  LRU  or  SRU  difficult  Furthermore,  because  electronics  are 
sensitive  to  environmental  conditions  sudi  as  temperature  and  vibration,  die 
evidoice  of  dieir  fiulure  may  be  transienL  The  ability  to  isolate  die 
malfunctioning  components  may  be  very  difficult  in  the  relatively  quiescent 
environment  of  a  maintenance  shop. 

The  result  of  difficult  fiiult  isolation  is  an  increased  number  of  fidse  removals  (Le., 
no  evidence  of  failures  [NECH^]).  Moreover,  tedinidans  in  seardi  of  a  fiiult  may 
remove  and  test  a  series-or  "diain''-of  several  related  LRUs.  The  more 
"related"  one  LRU  is  to  anodier  in  die  weapon  system,  die  more  likely  it  is  diat 
their  removals  will  be  positively  correlated.^  False  removals  and  chains  are 
especially  undesirable  because  diey  increase  support  costs  widuni  t  a  concomitant 
increase  in  weapon  system  availability. 

"Lemons" 

High-tech  systems  have  also  revealed  a  new  R&M  phenomenon,  die  emergence 
of  "lemon"  LRUs.  A  "lemon"  is  an  individual  LRU  that  exhibiis  dironic 
performance  degradations  that  are  not  common  to  other  components  of  die 
same  design.  Lemons  circulate  repeatedly  dirou^  die  maintenanoe  system, 
creating  an  enormous  burden  (dirough  repeated  removal  and  attempted  repair) 
while  contributing  very  little  to  operational  capability.  Evidence  from  RAND 
studies  conducted  for  die  Air  Force  (discussed  in  more  detail  below)  shows  diat 
specific  lemons  can  ctsisume  20  times  the  number  of  SRUs  as  do  weU-bdiaved 
LRUs  and  provide  only  a  fraction  (3  parent)  of  die  servke  tiine.  These  LRUs  are 
not  necessarily  Ixnn  bad  but  appear  to  become  fiiulty  over  time  for  reasons  diat 
are  not  yet  well  understood. 


’'Robbins  a991). 
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Examples  of  High-Tech  R&M  Problems 

The  four  R&M  problems  appear  to  be  eidemk  to  hi^>tech  s)rsteais:  That  is, 
they  are  not  characteristic  of  ttie  systems  of  only  (me  Service  or  anottter.  On  ttie 
one  hand,  this  means  that  the  Army  cannot  hope  whoUy  to  avoid  diese  problems; 
on  dte  other  hand,  it  means  that  ttie  Army  may  be  able  to  benefit  from  the 
experience  of  the  other  Services  and  apply  some  of  their  lessons  learned. 

For  well  over  a  decade,  die  Air  Force  and  RAND  have  worked  togedier  to 
acquire  a  great  dead  of  knowledge  about  the  R&M  characteristics  of  hi^-tech 
avicmics.  The  bulk  of  the  research  focused  cm  approaches  to  minimize  fault 
detection  and  fault  isolaticm  problems  and  to  improve  logistics  supportability  of 
fielded  systems.^  More  recent  RAND  analyses  of  advanced  Army  systems,  swdi 
as  the  MlAl  tank  and  the  Apache  helicopter,  evidence  R&M  challenges  diat  are 
fundamentally  die  same  as  diose  experienced  by  Air  Force  systems  in  die  mid-  to 
late-1980s.^  It  may  be  that  die  Army  can  benefit  fitrni  die  Air  Force's  experience 
and  previous  research  investments. 

The  next  section  presents  real-world  evidence  of  the  four  R&M  problems  drawn 
from  die  aforementioned  RAND  studies.  We  point  out  that  ncme  of  the  examples 
should  be  considered  unique  to  the  weapon  system.  Radier,  we  wish  to  show 
that  these  problems  are  endemic  to  hi^-tedi  ecpiipment  and  dierefore  typify  die 
R&M  performance  of  a  variety  of  high-tech  weapon  systems.  The  examples 
below  will  be  drawn  from  studies  of  die  Air  Fence  F-16A/B  aircraft  and  F-15C/D 
aircraft  and  the  Army  AH-64  helicopter  and  MlAl  tank. 

Experience  with  Fault  Detection  and  Reporting 

A  study  that  RAND  conducted  for  the  Air  Force  in  1984  as  part  of  die  F-15/F-16 
Radar  R&M  Improvement  Program  uncovered  an  interesting  set  of 
phenomena.^^  Operators  did  not  alwa^  report  a  symptom  of  a  fault  to 
maintenance;  and  when  they  did,  die  field  tedinidans  were  not  always  able  to 
duplicate  the  fault  and  diagnose  a  problem.  The  BIT  mi^t  have  detected  a  fault 
during  operation,  but  field  tedinicdans  were  unable  to  duplicate  ^nnptoms  of  die 
fault  and  consequendy  made  no  repair.  Field  technicians  may  have  removed  an 
LRU  and  sent  it  to  die  shop,  but  the  shop  technicians  could  find  no  evidence  of 


good  siuvey  of  ttw  work  done  in  diis  am  for  the  put  25  yeais  is  provided  in  unpublished 
resunA  by  (SebmanetaL,  on  lessons  for  future  avionics  fomn  research  with  the  Air  Force. 

^Ihe  MlAl  study  is  documented  in  Berman  (1988).  The  AH-S4  study  is  presented  in  Rdbbins 
(1991). 

l<^Obman(19e9). 
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fadluie  (ttius,  tt\e  LRU  was  considered  an  NEOF).  Steilarly,  shop  teduudans 
may  have  removed  an  SRU  from  an  LRU  and  sent  it  to  the  depot,  but  die  depot 
technicians  found  that  die  SRU  retested  okay.  Many  items  had  some 
maintenance  performed  <m  them,  consuming  lots  of  logistics  resources,  but  few 
repairs  were  made. 

Faults  Are  Not  Reported.  Figure  12  shows  the  events  surrounding  one 
F-16A/B's  low-power  radio  frequency  (LPRF)  unit  in  the  radar  system. 
Throughout  the  six-mondi  study  period  there  was  an  irregular  pattern  (rf  BIT 
indications  for  the  LPRF— die  first  two  flights  detected  something,  the  diird  fli^.t 
did  not,  the  next  four  flints  again  did,  die  next  flight  did  not,  etc.  This  pattern 
suggests  a  reason  for  die  reluctance  of  the  operator  to  file  a  report  and  illustrates 
die  inherent  limitations  of  die  BIT.^*  OveraU,  the  BIT  indicated  41  faulty  flights 
but  die  pilot  reported  only  two  to  maintenance.  The  first  fault  report  was  made 
in  July  after  12  previously  ignored  BIT  indications;  die  fault  had  deteriorated  to 
the  point  where  a  circuit  board  in  die  LPRF  started  generating  smoke  diat 
entered  the  cockpit.  After  die  LPRF  was  replaced,  diere  were  28  consecutive 
fault-free  flights.  Eventually  die  BIT  detected  29  additional  faults  before  the 
operator  reported  the  second  problem  to  maintenance  in  December.  The  bad 
LPRF  was  not  removed  and  remained  installed  on  die  aircraft  at  the  end  of  the 
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76 


SOURCE:  F-16A/B  Radar  MaturationalDeveiopfnent  Data  and  Analysis  Phase  1984-1985. 


Hgnie  2.2 — Example  of  Fault  Rcpoiting/Detecting  Problems  (F-16A/B) 


^^However,  the  BIT  system  for  die  F-16  generates  few  tabe  positive  indications,  lliat  is,  when  it 
detects  a  fault,  one  usually  exists,  even  if  the  BIT  does  not  detect  it  on  every 
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peiiod.^^  It  is  important  to  note  that  ttie  aircraft  at  ttiis  point  was  still  omsideied 
FMC  throughout  most  of  ttie  period  by  standard  reporting  systems  despite 
repeated  evidence  that  it  contained  a  faulty  con^>onent  While  dus  one  example 
of  the  fault  reporting  problem  is  more  extreme  than  tibe  typical  case,  the  bottom 
line  is  that  for  every  five  iiulications  of  a  fault,  whedwr  detected  by  ttw  BIT  or  dte 
operator,  only  one  was  reported  to  maintenance  for  repair. 

Maintenance  Technicians  Need  History  of  Fault  Indications.  Wluleitmi^tbe 
asstuned  toat  the  LPRF  was  performing  up  to  standard  on  toe  76  flights  toat 
were  fault-free  from  the  BIT'S  perspective,  a  review  of  toe  pilot's  observations 
provides  a  different  picture.  As  Table  2.1  shows,  at  times  toe  operator  noted 
degraded  performance  toat  was  not  picked  up  by  toe  BIT.  After  the  rrew  LFRF 
was  installed  in  July  (Episode  A),  toe  operator  reported  a  problem  to 
maintenance  on  only  two  occasums,  one  of  which  had  not  been  BIT  detected. 

It  appears  that  fixing  toe  LPRF  problem  was  inadequate  in  part  because  the 
maintenance  technicians  were  deprived  of  toe  history  of  the  BlT-detected  faults 
and  operator  observations.  Table  2.1  shows  toat  toe  lack  of  this  information 
precluded  the  proper  corrective  actions.  The  colurxm  labeled  FMC  indicates 

Table  U 

Weapon  System  Opentois  Do  Not  Always  Request  Maintenance  Actions 
When  They  Observe  Fa^ts 

(data  from  ndar  in  F-16A  aiioaft  no.  0^  gathered  during 
F-16  A/B  Radar  Maturation  Development  Data  and  Analysis  Phase  1984-1985) 


Date 

Problem  Observed 

BIT  Indication? 

FMC? 

July  17 

Lock-on  problem 

- 

Yes 

July  19 

Smedte  in  ooc)q>it 

Yes 

- 

July  23 

Lock-on  problem 

Yes 

Yes 

July  26 

Lock-on  problem 

- 

Yes 

July  26 

Degraded  perfonnanoe 

- 

Yes 

Aug  10 

Lock-on  problem 

- 

Yes 

Aug  13 

False  targets 

Yes 

Yes 

Sept  10 

False  targets 

- 

Yes 

Sept  11 

False  targets 

- 

Yes 

Oct  15 

False  targets 

Yes 

Yes 

Oct  16 

False  targets 

Yes 

Yes 

Oct  26 

False  targets 

- 

- 

Nov  26 

Degraded  performance 

Yes 

- 

Jan  10 

Degraded  perfomumce 

- 

Yes 

At  Ae  contractor's  request,  die  LPRF  was  eventually  removed  from  die  aircraft  in  )anuary 
1985  and  tested  by  the  contractor. 
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whether  ttie  operator  reputed  the  weapon  system  to  be  FMC.  On  October  26, 
the  operator  reported  a  need  for  maintenance  because  of  numerous  false  targets 
even  thou^  the  BIT  did  not  detect  a  foult  The  maintenance  tedmidans, 
unaware  of  d\e  BIT  history,  conducted  BIT  ground  tests  tttat  also  showed  no 
evidence  of  failure.  The  technicians  took  no  further  action  and  instead  waited  to 
see  if  the  operator  asked  for  maintenance  again,  which  he  did  after  numerous 
flints  with  BIT  indications  (not  shown).  Again,  dtey  executed  the  BIT  on  the 
ground  and  found  no  indication  of  a  fault.  They  again  diose  to  take  no  furdier 
action. 

We  speculate  ttiat  if  the  mainteiunce  technicians  had  seen  the  history  of  fault 
indications,  it  is  unlikely  they  would  have  allowed  Ae  LPRF  to  remain  in  die 
weapon  system  as  long  as  diey  did.  Thus,  their  performance  was  not  an  instance 
of  poor  maintenance,  nor  can  die  operator  be  blamed  for  poor  fault  reporting. 
Radter,  this  series  of  episodes  illustrates  the  inadequacies  of  die  established  RAM 
approach  to  meet  the  challenges  presented  by  hi^tech  failure  modes.  The 
established  approach  relies  on  (1)  a  data  system  dtat  collected  indications  of  a 
fault  when  an  operator  believes  it  is  severe  enough  to  call  for  maintenance  and 
(2)  maintenance  procedures  that  suggest  a  s)rstem  is  fully  mission  capable  if  its 
faults  cannot  be  duplicated  in  a  ground  test  environment  The  standard  data 
collection  system  is  simply  not  comprehensive  and  integrated  enough  to  detect 
die  faults  that  were  occurring  and  to  aid  in  the  isolation  of  the  problem.  The 
analysis  presented  here  relied  upon  die  research  team's  special  data  collection 
effort. 

Fault  Reporting  Increases  When  Operators  Face  Combat  Table  2.1  provides 
evidence  that  operators  may  frequendy  report  a  weapon  system  to  be  FMC  for  an 
imminent  mission  even  diough  diey  have  noted  hiults  and  degraded 
performance.  Moreover,  as  Figure  23  suggests,  even  in  training,  operators 
increase  their  rate  of  fault  reporting  when  faced  widi  a  combat-related  mission, 
reflecting  their  increased  concern  with  die  mission  capability  of  their  weapon 
systems.  The  figure  reports  removals  of  the  Apadie  helio^iter  Mission 
Equipment  Package  (MEP),  the  Target  Acquisition  and  Designatkm  Si^t  (TADS) 
electronics  unit  (TEU),  and  the  laser  transceiver  unit  (LTU). 

Ongoing  RAND  studies  of  the  Army's  and  Air  Force's  experiences  in  Operation 
Desert  Shield/Storm  provide  additional,  diough  preliminary,  evidence  diat  fault 
reporting  rates  increase  when  weapon  system  operators  are  faced  with  combat 
missions.i3  It  appears  diat  the  number  of  hiults  (Le.,  repair  requests)  reported  for 


^^Gdxnan  et  al.,  unpidilished  RAND  reicwth  on  finding  and  fixing  bad  actor  avionics. 
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Fignfc  23— Apadw  H%li*Tcdi  Campoatnt  Rtmoval  Rates  for  OimlMl>lldalcd 
Miaeiona  Exceed  Thoec  foe  Otticr  Tiaining  Miaakiii  Tjrpcs 


Air  Force  and  Army  hi^tedt  coB^Kxients  were  more  ttian  twice  ttie  peacetime 
values.  When  given  a  dioke  as  to  ^ledter  a  system  is  mission  capable  or  not, 
die  operators  made  a  very  different  choice  ^ten  faced  with  combat  Thus,vhat 
appears  as  a  maintainability  problem  is,  in  fact,  a  mission  reliability  problem. 

Experience  urith  Fault  Isolation 

Maintenance  Tcduiidans  Sec  No  Evidence  of  FaUvre.  Figure  2.4  illustrates  the 
problem  wife  fault  isolation  by  depicting  the  fouh  isolation  efficiency  associated 
witti  27  types  of  subsystems  on  die  F-16A/BaircrafL  Ihe  steeper  die  slope  of  the 
line  associated  with  a  qMdfic  weapon  system,  the  higher  die  fault  isedation 
efficiency.  Forevery  100  flints,  16  subsystems  had  one  or  no  repair  actions.  The 
fire  conticd  system  averaged  nine  leciueds  for  repair  Half  were  repaired  and 
half  were  not  because  maintenanoe  could  not  isolate  die  problem  (eidier  the  BIT 
could  not  duplicate  die  malfiinction,  or  die  teduiidans  could  not  duplicate  what 
the  operator  saw).  In  odier  words,  die  efficiency  of  fault  isolation  at  the  flight 
line  was  about  50  percent 


.9  .8  .7 


UndkisSMT 


1/ 

f 

oonipol 

1  /  EiaeL  •  y 

^  ECM 

•  •  PoBw  y  jT 

/Xaux.p^ 

16 

SyeiNna  ^ 

Number  not  rapaiied 

SOURCE:  F-16  AA  iraMmanM  (Mt  ootadion,  1985. 
FjgmelA-TO^MJneFMltltoirtian  Efficiency  (F-15  A/8)  per  too  PUghti 

Figure  2.4  also  illustrates  how  mudi  easier  it  is  to  iscdate  faults  in  medunical 
equipment  compared  wi8ihig^v4edicqui{anent  Note  ttuit  limiting  and  landing 
gear,  botti  medianical  cooqianents,  have  fault  isolation  efficiencies  near  80 
percent  on  the  flight  line. 

This  research  also  tracked  fault  isoiation  efficiency  beyond  die  flight  line  (not 
shown  in  Rgure  2.4).^^  For  die  radar  (fli^t  control),  51  percent  of  the  reported 
hiults  were  repaired  on  the  flight  line.  The  inteimediate  shop  fixed  68  percent  of 
the  radars  it  saw,  and  die  depot  repaired  80  percent  of  the  radar  SRUs  it  saw.  On 
die  mrfece,  some  of  diese  efficiencies  might  q’pear  acceptable,  but  a  closer 
examination  indicates  diat  die  overall  fauh  isolation  efficiency  for  die  radar  is 
just  28  percent  hi  odier  words,  about  one  in  four  requests  for  maintenance 
resulted  in  a  repair  action.^ 


^*Tlwwwrdiw»  bud  OB  a  ipcdil  date  opBecaoncfhrtinvotving  16.000  flighli  of  the  F-16 
in  1984-1985. 

overall  cCliciency  late  of  .28  ii  darivod  from  multiplying  the  cfBdendei  at  aadi  of  the 
three  levris:  31  x  48  x  JO. 
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GckmI  LRUs  Are  Removed  to  Isolate  die  Bad.  Difficult  fault  isdation  creates  an 
additional  nuintenance  burden  because  technicians  may  mistalcaily  replace 
good  LRUs.  Tabte  Z2  shows  one  aircraft's  avionics  LRU  replacements  for  a 
three-month  period.  Note  that  on  September  15,  maintenance  technicians 
replaced  the  four  radar  system  LRUs  and  die  electronics  unit  of  die  heads-up 
display  (HUD)  in  response  to  a  reported  fault  The  transmitter  continued  to  be  a 
problem  and  was  replaced  four  more  times.  All  told,  the  transmitter  was 
replaced  five  times,  ten  times  more  dum  the  expected  replacement  rate  of  0.5. 
(And  die  transmitter  still  may  not  have  been  die  culprit) 

The  Army  has  experienced  similar  fiiult  isolation  problems  widi  die  MlAl  tank 
and  the  AH-64  Apache  helicopter.  RAND  analysis  of  Army  data  uncovered  that 
the  MlAl's  fire  control  system  was  experiencing  chains  (removals  of  multiple 
boxes  to  isolate  a  fault  in  one)  in  27  percent  of  its  removals,  with  an  average  chain 
length  of  three  LRUs.^^  The  early  fielding  of  the  Apadie  fleet  sufiered  to  a  great 
degree  from  die  same  maintainability  problems  as  odier  hi^tech  weapon 
systems.  Poor  fault  detection  was  evidenced  by  a  40  percent  error  rate  in  BIT- 
detected  faults.  Poor  fault  isolation  resulted  in  hig^  NEOF  rates:  25-30  percent 
for  its  MEP  LRUs,  and  29  percent  for  diose  in  die  TADS/Pilot  Ni^t  Vision 
Sensor  (PNVS).  Some  high-driver  LRUs  had  NEOF  rates  as  hi^  as  50  percent  at 


Table  2.2 

Inadequate  Diagnoetica  Create  Pmblcma  in  Sustaining  Mission-Effective  Avionics 
(removals  of  avionics  LRUs  from  F-16  A/B  no.  0021) 


Date\ 

LRU 

Fli^t 

Control 

Panel 

Electronic 

Computer 

Assembly 

Air 

DaU 

Computer 

Low-Power 

Radio 

Frequency 

Unit 

Trans¬ 

mitter 

Digital 

Signal 

IVoccssor 

Com¬ 

puter 

Electronics 

Unit 

Septa 

1st 

Sept4 

2nd 

S^15 

1st 

1st 

1st 

1st 

1st 

Sept  22 

2nd 

Sept  26 

1st 

Octl 

3rd 

Oct8 

2nd 

Oct  10 

1st 

Oct  26 

3cd 

Oct  27 

4dt 

2nd 

Nov  26 

5lh 

Tout 

1 

2 

1 

3 

5 

2 

1 

1 

Expected 

0.1 

0.2 

0.2 

05 

05 

0.3 

05 

05 

^^The  study  is  reported  in  Bennan  (1968),  diough  this  particular  finding  remained  unpublished. 
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FraiRucker,  the  Anny  Aviation  Center  and  SduxkL  Iheae  NEC^  resulted  from 
fault  isolation  probtems  at  the  flight  line.’^ 

Effects  of  Poor  Fault  Isolation.  These  examples  demonstrate  die  potential  kx 
adverse  effects  on  the  logistics  structure.  St^ipasedly  good  LRU  spares  provided 
to  field  maintenance  and  good  SRUs  provided  to  intermediate-level  maintenance 
are  still  potentially  faulty.  Furdwr,  die  handling  of  eadi  component  uses 
valuable  supply,  maintenance,  and  transpevtation  resources. 

What  does  poor  fault  isolation  mean  for  weapon  system  effectiveness?  Many 
faults  remain  in  the  system  unfixed.  Weapon  systems  might  be  repaired  widt  a 
defective  LRU  duit  had  malfunctioned  yet  showed  no  evidence  of  failure  to  the 
maintenance  technician.  Smilarly,  faulty  LRUs  might  t)e  "fixed"  widt  defective 
SRUs.  OveraU,  die  result  is  an  overstatement  of  available  weapon  systems  dtat 
can  deliver  their  full  designed  performance. 


Fault  Removal  Efftciency 

By  combining  diese  diree  elements  of  the  maintenance  process  (fiiult  reporting, 
detection,  and  isolation),  we  emerge  with  a  view  of  die  seriousness  of  die 
maintainability  problem.  The  F-16  study  found  diat  only  20  percent  of  die  fiiults 
were  reported  and  that  of  diose  only  28  percent  were  repaired.  This  means  that 
fiiult  removal  efficiency^*  is  a  mere  5  percent-Le.,  only  1  in  20  faults  are  rqmired 
by  maintenance. 

Low  fault  removal  efficiency  is  not  a  problem  peculiar  to  die  F-16A/B  radar.  A 
parallel  collection  of  engineering  data  far  the  F-ISC/D  radar  ofiers  anodier 
example.  In  this  case,  die  KT  system  hidicated  faults  in  one  of  every  duee 
flights.  When  die  BIT  detected  a  fiiult,  it  emreedy  isedated  the  fiiult  only  one- 
third  of  the  time  and  misdiagnosed  die  odier  two-diirds  (eidier  no  fault  existed 
or  die  BIT  identified  die  wrong  LRU  as  die  finihy  component).  It  also  failed  to 
detect  faults  diat  actually  existed  in  another  one-third  of  the  flints.  As  a  result. 


is  our  undemanding  that « laige  cflioft  on  the  part  of  the  Aimy  led  to  a  maitod 
toiprovement  in  this  aspect  of  the  Apache's  perfonnance.  The  Army  estaUidied  the  Apadie  Action 
Team  in  the  eaity  field^  stage  to  in^MOve  the  rdiability  and  maintainability  of  the  hoioopter. 
Hotvever,  impnwremenisimeimplemenled  after  the  bulk  of  the  fielding  and  oDneentrated  on 
mechanical  parts. 

^*Fault  removal  efficiency  is  a  measure  of  total  mahttenanoe  system  peffocmance.  Itisdefined 
as  the  percentage  of  fsutes  at  the  end  hern  removed  by  replacenient  of  fsuhycMnpooents.  bia 
multiechekin  maintenance  system,  the  perfoimance  of  each  eehekmconliibutes  to  the  system's  fault 
removal  efficiency. 

^^AddhionalW,  a  set  of  special  studies  was  done  on  the  electronic  countenneasurefECM) 
systems  for  the  F-15  C/D  and  the  F-16  A/Band  similar  lesullB  were  foimd.  Thew  studies  are 
reported  in  Gebmanetal.  unpublished  RAND  research  on  reliability,  maintafatobility,  and 
maintenance  of  electronic  oourdermaasures  for  tactical  air  warfare. 


20 


BIT  system  was  not  believed.  Faults  remained  in  ttw  system  for  a  number  of 
flights  before  repair  was  made.  For  20  percent  of  dw  LRUs  sent  to  ttw 
intermediate  shop  for  repair,  dte  shop  implemented  repair  actions  duit  were 
unrelated  to  the  faults  that  occurred  in  flight  Moreover,  85  percent  of  the  faults 
identified  had  been  identified  previously. 

Figure  2.5  shows  the  seriousness  of  die  maintainability  problems  for  die  F-15  and 
F-16  radars.  The  figure  shows  diat  of  all  indicated  faults,  relatively  few  are  new 
faults;  most  are  old  faults  that  have  been  previously  detected  but  not  removed.^ 
The  persistence  of  old  faults  implies  diat  the  R&M  challenge  of  high-tech  systems 
is  predominandy  a  problem  with  maintainability  radier  than  reliability.  Old 
faults  gready  increase  die  burden  <m  the  logistics  structure  and  gready  reduce 
the  availability  of  FMC  weapon  s)^tems. 


IS 

■§» 


8 


& 


£ 

s 

8 


£ 

% 

16 


F-15  C/D  F-16A/B 


NOTE;  F-15C/D  Radar  and  F-18A/B  Radar  MaluiaMonalDtveloptnant  Data  and  Analysis 
Phase  1984-19B5. 


Figure  2J— Old  Faults  (Le.,  Maintainability  Problems) 
Domiiiate  New  Faults  (Rdiabilily  Problema) 
<F-15C/D  radar  and  F>16A/B  radar) 


^^^Some  faults  are  "reinovied*  through  a  maintenance  action-such  as  raetting  a  button-other 
than  removal  of  a  suspected  faulty  component 


Experience  unth  Lemons 

A  small  subset  of  high-tech  subsystems  exhibit  tiie  lemon  problem:  Particular 
LRUs  show  failure  aiul  degraded  performaitce  rates  that  are  starkly  hi^ier  ttum 
those  seal  in  other  items  of  identical  design.^^  Aldiou^  smne  evidence  suggests 
that  the  incidence  of  lemons  rises  with  the  electronic  conqilexity  of  the  LRU's 
design,  one  study  found  that  approximately  9  percent  of  any  given  high-tedi 
LRU  type-for  example,  the  transmitter  in  a  radar-exhibit  this  behavior.^  Their 
differing  failure  modes  may  be  triggered  by  particular  characteristics  of  the 
environment  and  may  not  be  present  in  test  bench  environments.  Because  these 
problems  do  not  occur  in  the  normal  maintenance  test  environment,  diey  can 
cause  repeated  removals  wittiout  (attempted)  repair  removing  die  cause  of  the 
problem. 

An  Air  Force  study  done  in  1985  and  1986  on  the  F-15C/D  radar  unit  provides  an 
example  of  the  lemon  problem.  Table  23  shows  the  number  of  individual  radars 
that  made  a  large  number  of  trips  to  the  intermediate  repair  shop.  Duringdie 
threc'month  exercise,  programmable  signal  processor  #1059  visited  die  diop  12 
times,  and  transmitter  #0067  visited  the  shop  10  times.  These  observations  led  to 
a  special  data  collection  at  Bitburg  in  1984  to  more  fully  define  the  problem. 

As  expected,  some  specific  units  were  removed  more  fiequendy  than  odieis. 
Figure  2.6  shows  the  radar  digital  processors  diat  were  removed  and  replaced 
four  or  more  times  during  the  year.  On  average  across  the  fleet,  diis  type  of  LRU 
was  removed  1.9  times  during  die  year  and  0.6  times  per  100  flying  hours. 
Particularly  noteworthy  are  the  five  digital  processors  diat  had  to  be  replaced 
more  than  15  times  per  100  flying  hours.  These  five  are  considered  candidate 
lemons  and  subject  to  furdier  analysts. 

The  implications  on  the  field  maintenanoe  activity  are  obvious.  Technicians  had 
to  take  many  repair  actions  diat  provided  very  little  value  in  terms  of  weapon 
system  availability.  These  lemon  LRUs  amounted  to  about  9  percent  of  the 
processors  but  provided  only  3  percent  of  die  flying  hours. 

The  implications  for  die  entire  logistics  structure  are  even  greater.  Ti^k  2.4 
shows  that  the  lemons  had  a  mean  time  between  removal  of  four  hours  and 
accounted  for  one-third  of  die  intermediate  shop  returns.  Even  worse,  diey 

^^The  problem  of  lonans  among  weapon  systems  and  weapon  system  ooniponnls  was  first 
investigated  by  RAND  in  the  early  1960s  (W.  R  McGlottdin  and  T.  S.  Donaldson,  1964,  and  T3. 
Doiuldson  and  AJ’.  Sweetland,  1966).  Some  other  RAND  leseaidi  often  refers  to  lemons  as  *lMd 
actors." 

^^Gcbman  et  aL,  unpublished  RAND  research  on  finding  and  fixing  bad  actor  avionics. 
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Table 

Some  Specific  Componenta  Make  Many  Tripe  to  ttie  Shi^ 


Number 
of  Visits 
perUrtit 

Number 
of  Units 

Uitit  Type  arul  Seriai  Number 

12 

1 

PrograiTuitable  sigtud  fmweasar  *1(S9 

10 

1 

Transnitter  #0067 

9 

1 

Programmable  sigiutl  proceastM’  *1015 
Traitsmitter  #0525 

8 

4 

Receiver  #0471 

Arudog  signal  processor  #0564 
Programmable  signal  processor  #1057 

7 

1 

Antenna  #06S5 

6 

7 

Various  units 

5 

17 

Various  units 

NOTE:  F-15  radar  units  at  Biibuig,19M. 


0  5  10  15  >15 


Removais/100  flying  hours 

FigaK2.A — ^Lemon  Digital  ProccasonWeR  Replaced  More  ttan  15  Times  per  100 

Flying  Hows 
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Tabic  2.4 

Radar  Lemons  Consainsd  20  Times  Mok  SRUs  than  Did  Other  LRUs 


Good 

UUs 

Lemon 

LRUs 

AU 

LRUs 

Total  visits 

768 

382 

1150 

Mean  time  between  removal 
(MTBR)  (hours) 

34 

4 

24 

NEOF  rate 

40% 

35% 

39% 

Reseat  and  adjust 

24% 

24% 

24% 

Remove  and  replace 

33% 

34% 

33% 

LRU  not  reparable  diis 
station  (NRTS) 

1.7% 

03% 

1.3% 

SRUs  consumed 

362 

181 

543 

SRUs  cemsumed  per 

1000  flying  hours 

2.1 

44.0 

3.1 

coiuumed  20  times  the  SRUs  consumed  by  good  LRUs  for  every  1000  flying 
hours.  That  amounts  to  a  50  percent  increase  in  die  number  of  SRUs  sent  to  the 
depot  for  repair. 

The  implications  of  lemons  for  weapon  system  effectiveness  are  also  serious.  The 
lemon  problem  does  not  appear  to  be  limited  to  only  one  LRU  for  a  given 
weapon  system,  or  even  for  a  given  high>fech  subsystem.  Each  type  of  high-tech 
LRU  appears,  from  our  studies,  to  be  subject  to  lemon  units;  all  eight  hi^tech 
LRUs  in  the  F-15  radar  have  lemons.^  In  an  F-16  fleet,  if  loncm  LRUs  are 
randomly  distributed  among  die  weaptm  systems,  diere  is  a  55  percent 
probability  diat  any  given  weapon  has  at  least  one  lemon  LRU  in  it 

Although  litde  is  known  about  their  causes,  our  observations  of  die  Air  Force 
experiences  indicate  diat  these  LRUs  are  not  manuhictured  as  lemons  but  become 
sick  over  time.  Detailed  engineering  analyses  of  a  number  of  the  isolated  LRUs 
found  that  diese  units  suffered  probtems  under  environmental  stresses,  Le., 
thermal  or  vibration,  diat  were  not  detectable  cm  benign  test  benches. 

While  we  have  used  exanqiles  of  USAF  systems  to  describe  the  lemon  problem 
and  its  ramifications,  we  do  not  mean  to  imply  diat  only  Air  Force  systems  are 
susceptible.  Rather,  we  have  had  a  greater  c^iportunity  to  study  this  problem  in 
Air  Force  systems,  and  the  Air  Force  has  generally  made  greater  use  of  complex 


^Other  high-tech  systons,  like  the  ECM  systenv,  in  both  the  F-15  and  F-16  appeir  to  have 
similar  lemon  units. 
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high-tech  components.^^  As  yet.  Army  weapon  systems  have  not  incorporated 
the  level  of  complexity  that  is  more  commonplace  in  the  Air  Force,  but  tihat  is 
higlily  likely  to  change  over  time.  Thus,  we  would  predict  that  current  Army 
systems  do  not  generally  suffer  from  a  lemon  problem  to  die  same  degree. 
However,  Figure  2.7  shows  signs  of  just  such  a  problem  emerging,  and  it  is  likely 
to  grow  far  greater  as  the  Army  uses  mote  high-tech  components  and  their 
complexity  increases.  The  figure  shows  the  distribution  of  lemons  across  LRUs 
of  one  of  the  more  sophisticated  Army  subsystems,  the  Apache  TADS/PNVS.  In 
the  TADS/PNVS,  one  LRU,  the  TEU,  is  by  a  large  margin  the  most  complex  piece 
of  electronics  along  the  dimensicms  noted  above.  And,  unsurprisingly  in  view  of 
the  Air  Force  experience,  that  LRU  shows  strong  signs  of  lemon-like  behavior. 

Yet  even  the  TEU  is  simple  technology  compared  with  what  the  Army  intends  to 
buy  in  the  future.^ 


12  I - 1 


Apache  TADS/PNVS  LRUs 


Figure  2.7 — Lemons  May  Be  Emerging  in  Army  Weapon  Systems 


^^High-tedi  electronics  are  becoming  increasingly  complex,  measured  along  three  dimensions: 
the  number  of  circuit  cards  in  a  box,  the  density  of  tlw  circuits  on  the  cards,  and  the  interconnections 
between  the  cards  (and  to  other  components). 

^Our  RAND  colleague  Marc  Robbins  provided  us  with  this  example  of  lemon  behavior  in  an 
Army  weapon  system. 
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Basic  fault  reporting,  detection,  and  isolaticm  defickndes  further  complicate  the 
lemon  problems.  We  noted  earlier  how  die  deficiencies  in  the  automatic  fiiult 
detection  and  fault  isolation  procedures  have  led  to  a  lack  of  confidence  in  these 
systems  by  operators,  which  have  led  them  to  ignore  some  of  the  mdkaticms  of 
faults.  We  also  noted  how  the  lack  of  total  reporting  of  all  indications  of 
malfunctions  have  led  to  difficulty  in  fault  isolation.  This  results  in  as  many  as 
one-half  of  the  units  being  returned  to  die  system  as  serviceable  widi  no  repair 
made.  Additionally,  because  lemon  LRUs  exist,  some  of  the  "repairs"  diat  are 
made  have  no  relationship  to  the  cause  of  die  problem.  Our  data  indicate  diat  as 
many  as  20  percent  of  the  repairs  that  are  made— i.e.,  remove  and  replace  an  SRU- 
"fix"  components  that  are  not  broken. 


Operational  Effects  of  R&M  Problems 

R&M  problems  that  are  not  properly  detected,  reported,  isolated,  and  removed 
not  only  create  a  burden  on  die  maintenance  system;  they  also  have  deleterious 
and  potentially  dangerous  effects  on  operational  planning  and  capability.  If 
weapon  system  availability  rates  are  overstated  by  the  present  data  collection 
and  analysis  methods,  then  planning  factors  for  weapon  system  availability  are 
probably  incorrect  and  planning  factors  for  weapon  system  support 
requirements  are  probably  underestimated. 

There  is  also  evidence  that  weapon  system  operators  become  aware  of  the 
limitations  of  the  maintenance  system  to  remove  faults  and  adjust  their  own 
behavior  accordingly.  Many  fault  indications  are  not  reported  by  operators 
apparently  because  they  are  aware  of  die  difficulty  that  maintainers  face  in 
isolating  and  removing  faults.  This  leaves  the  unfortunate  situation  in  which 
components  exhibiting  indicated  faults  remain  in  weapon  systems.^  Of  course, 
operators  may  change  their  behavior  depending  on  the  mission  they  are  about  to 
conduct.  While  fiiult  indications  are  frequendy  ignored  in  peacetime,  they  are 
not  ignored  in  war.  Even  during  exercises  diat  required  great  precision, 
operators  report  more  faults  than  during  normal  peacetime  training  operations.^ 
Preliminary  analysis  of  data  about  Operation  Desert  Storm  indicates  that 
demand  rates  for  some  components  in  some  cases  were  twice  the  peacetime 
rates.  Even  in  peacetime,  demand  rates  vary  by  mission  t)rpe.  As  Figure  2.4 


^^This  is  especiaUy  so  widt  the  F-16  radar  for  which  RAND,  tiwou^  a  special  study,  found  that 
the  BIT  system  is  reliable  when  it  indicates  a  fault  In  at  least  95  percent  of  the  cases,  whim  die  BIT 
detected  a  foult,  a  foult  existed.  But  pilots  did  not  rdiably  request  maintenance. 

^Studies  of  Coronet  Warrior  and  ECM  test  programs  revealed  a  higher  demand  rate  than 
normal  peacetime  rates. 
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showed,  removals  of  ^>adie  MEP  LRUs  incicaae  in  more  demanding,  combat- 
type  missions. 

Summary  and  Implications 

Hie  characteristics  of  high-tech  weapon  systems  indicate  a  need  for  a  new 
approach  to  R&M  onaruigement  Because  new  we^xms  and  maior  upgrades  use 
more  high-tech  components,  the  problem  will  only  get  worse.  Theieseardi 
already  done  in  ttiis  area  shows  that 

1.  Newweaponsystems,  or  nuijor  upgrades  with  high-tech  subsystems,  will 
have  R&M  problems  when  they  are  fielded. 

2.  Although  there  is  always  room  for  unprovement  in  logistics  reliability,  there 
appears  to  be  greater  room  for  improvement  in  die  design  of  die 
maintainability  of  die  new  weapon  systems. 

3.  The  maintainability  characteristics  are  complex  because  some  "failures*  are 
difficult  to  detect  and  the  ability  to  repair  diese  failures  is  elusive. 

4.  These  maintainability  improvements  will  require  design  dianges,  which 
should  be  incorporated  into  die  design  as  soon  as  passible. 

5.  Detection  and  isolation  of  design  defidendes  must  generally  rely  on  a 
significant  amount  of  information  from  s)r5tems  in  an  operational 
environment  for  analysis.  This  information  is  best  obtained  during  die 
initial,  low-rate  production  phase  for  two  reasons:  first,  because  diat  is  the 
first  time  diat  information  on  a  production  model  in  operation  is  available; 
and  second,  because  corrections  to  defidendes  found  can  be  implemented  in 
models  produced  during  full-rate  production,  avoiding  die  possibly 
prohibitive  expense  of  retrofitting  die  entire  line. 

6.  Even  after  correcting  the  design  defidendes,  a  number  of  individual  lemons 
will  exist  that  need  to  be  identified  and  managed  sq>arately. 

The  first  four  elements  are  primarily  observations  diat  need  to  be  conveyed  to 
managers;  the  last  two  elements  call  for  die  development  of  a  new  methodology. 
Section  3  of  this  report  describes  sudi  a  mediodology:  the  "maturation 
development  process." 


27 


3.  The  Maturation  Development  Process 


Faced  with  the  R&M  challenges  posed  by  fielded  hi^-tedt  weapon  systems,  the 
Services  have  developed  various  innovative,  albeit  ad  hoc,  ways  to  cope  witft 
them.^  Chir  focus  here  is  on  advocating  a  systematic,  research-supported 
approach  to  reduce  d\e  problems  for  existing  weapon  systems  and  for  high-tedi 
weapon  systems  now  being  developed  or  modified. 

As  noted  in  die  examples  dted  in  Section  2,  many  of  die  iruintainability 
problems  uncovered  in  previous  RAND  research  were  identified  diroug^  an 
intensive  data  collection  and  arudysis  effort  Those  analyses  identified  major 
fiiult  reporting,  fault  detection,  and  fault  isolation  problems,  as  well  as  the 
existence  of  the  lem<m  problem.  The  success  of  diese  analyses  suggests  a 
maturation  approach  that  involves  using  special  data  collection  and  analysis 
efforts-different  in  land  and  degree  from  what  programs  typically  have  done-to 
enable  weapon  system  sustainment  managers  to  identify  die  need  for  specific 
engineering  changes  and  improve  both  sustainment  and  weapmi  system 
capability. 

However,  each  of  the  examples  of  Service  experience  illustrates  applicaticms  of 
only  selected  elements  of  the  maturation  development  process  and  only  to 
fielded  systems.  None  exemplifies  application  of  the  full  process  to  a  new 
weapon  system.  The  purpose  of  diis  section  is  to  provide  a  detailed  description 
of  the  full  maturation  development  process  as  applied  to  a  new  weapon  system. 
The  section  argues  the  need  for  such  a  process,  outlines  its  phases  over  die  course 
of  a  system's  life  cycle,  and  explains  how  the  adoption  of  such  an  approach 
would  alter  or  augment  the  activities  currently  undertaken  in  each  stage  of  the 
weapon  system's  development  and  fielded  life. 


^^During  Operations  Desert  Shield/Stonn,Annystq>port  for  the  Apadie  was  innovative.  The 
levelof  intermediate  svqpport-intenns  of  TMDE— was  160  percent  of  doctrinal  ievds.  Depot-level 
support  was  provided  in  the  Continental  United  States  (C(X4US )  and  also  in  theater  in  ^  form  of 
sp«ial  repair  activities  (later  called  forward  repair  activities-FRAs)  and  contractor  siqiport,  as  well 
as  in  CONUS.  The  rapid  transportation  system  ttiat  was  developed  both  intertheater  ('Desert 
Express')  and  intradinter  ('Ciunel  Flints')  hdped  to  furttier  inqnove  the  availabUity  of  these 
%veapon  systems.  These  adaptations  rdlect  bodi  the  ability  to  respond  to  unusual  situations  and  the 
need  to  treat  high-tecdisystenu  differently.  Studies  need  to  be  conducted  to  determine  the  exact 
edndiveness  of  these  adaptatkms.  See  Robbins  and  Mciver,  unpublished  RAND  research. 
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Need  for  a  Maturation  Development  Process 

The  higji'iisk  R&M  characteristics  of  hi^tech  s)«teins  reqvtire  a  new 
incremental  process  for  maitaging  ttw  develc^nnent  and  sustainment  of  R&M 
over  die  life  cycle  of  a  complex  weapon  system.  Maturation  develt^mient  is  a 
OHitinuing  process  to  deal  e^ectively  widi  die  anomalies  in  the  R&M 
performance  of  the  weapcm  system. 

Objectives  of  Maturation  Development 

The  goal  of  maturation  development  is  the  delivery  of  full  design  performance  of 
the  weapon  system  under  mission  conditiMis  and  die  rapid  restoration  of  full 
design  performance  when  deficiencies  exist  Maturation  development  provides 
for  early  resolution  of  systemwide  R&M  problems  and  identification  of  lemon 
LRUs  that  make  their  way  to  die  field.  Early  identification  and  resolution  of 
R&M  problems  can  be  a  hi^y  effective  means  of  controlling  the  cost  drivers  of 
readiness  and  sustainment  and  of  managing  die  support  cost  of  the  weapon 
system. 

The  potential  benefits  of  in^lementing  sudi  a  concept  are  twofold:  first,  the 
achievement  of  full  designed  system  perfcmnance,  whidi  cimtributes  to  genuine 
mission  capability;  and  second,  the  reduction  in  support  costs  over  the  life  cycle 
of  the  system.  Setting  performance  considerations  aside,  these  cost  savings  are 
more  dian  sufficient  to  offeet  the  costs  of  implementing  die  maturation 
development  process  if  undertaken  early  enough  ti>  avoid  a  major  retrofit  of  an 
entire  production  line. 

Overview  of  the  Process 

Maturation  development  calls  for  ccmtinumis  data  collection  and  analysis  on  die 
R&M  performance  of  a  weapon  system.  For  new  weapon  systems,  data 
collection  begins  in  the  phases  preceding  fielding  and  intensifies  during  die 
period  of  initial  fielding.  Maturation  development  could  also  be  applied  to 
fielded  systems,  with  modifications.  In  such  a  case,  die  period  of  intense  data 
collection  and  analysis  would  be  implemented  in  corqunction  with  a  system 
upgrade. 

The  purpose  of  the  period  of  intense  data  collection  and  anal)rsis  would  be  to 
detect  and  isolate  design  deficiencies  by  intensively  operating  the  components  in 
the  environment  widiin  whidi  diey  will  normally  operate.  The  period  of 
maturation  development  coinddes  with  what  is  now  known  as  the  period  of 
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initial  (^leratianal  test  and  evahiatian.  It  would  invcdve  primarily  devek^nnent 
testing  witii  direct  partidpation  by  contractor  engineoing  personnel,  followed  by 
a  detailed  aiudysis  of  the  data  gatiwied.  During  tius  period  the  Army  woniki  be 
"measuring  operational  experience,  organizing  and  recording  R&M-related  data, 
interpreting  the  data,  and  drawing  ccmclusions  about  the  root  causes  of  the 
dcnniiumt  R&M  problems  tiuit  are  respcmsible  for  any  shortfalls  in  needed  R&M 
characteristics."^  The  output  of  this  dedicated  phase  feeds  the  engineering 
process  to  facilitate  changes  to  the  full-rate  production  versions  of  the  high-tech 
components  so  that  tiie  major  R&M  problems  are  resolved. 

These  activities  of  intensive  weapon  system  operation,  data  collection,  analysis, 
and  subsequent  engineering  improvemetts  must  be  preplanned  and 
incorporated  into  the  acquisititm  process.  While  tiw  plaiming  must  occur 
throughout  the  entire  acquisition  process,  tiie  bulk  of  tiie  operation  and  data 
aiudysis  activities  would  occur  during  the  latter  portion  of  tite  engineering  and 
manufacturing  development  phase,  Le.,  tiw  low  rate  of  initial  production  (LRIP) 
period.  LRIP  is  defined  by  law  to  be  that  period  preceding  the  full-rate 
production  during  which  (1)  the  production  base  is  established,  and  (2) 
operational  test  and  evaluation  of  the  weapon  system  is  conducted  to  verify  the 
operational  effectiveness  and  suitability  of  the  system. 

The  database  capabilities  set  up  in  the  exAy  acquisition  phases  and  used 
extensively  during  LRIP  would  ttien  be  used  in  a  much  more  modest  fashion 
during  production  and  fielding  in  order  to  analyze  performance  and  to  identify 
and  isolate  lemon  LRUs. 

A  necessarily  more  limited  version  of  tiie  maturation  develc^nnent  concept  could 
be  adopted  for  systems  that  are  already  fielded  and  for  which  only  a  limited 
database  exists.  These  systems  ate  likely  to  still  suffer  the  lack  of  maturity  one 
would  desire  in  a  hig^-tech  weapon  system.  Indications  of  an  immature  system 
include  a  high  degree  of  difficulty  in  fiiult  detection  and  fault  isolation,  along 
with  a  set  of  LRUs  tiiat  consume  large  amounts  of  logistics  resources.  Ihis 
approach  would  call  for  tiie  establishment  of  a  limited  database  to  identify  the 
extent  of  the  problem  and  to  isolate  the  most  immature  electronic  units.  A 
performance-oriented  system  for  tiie  tracking  of  equipment  repair  would  allow 
the  weapon  system  manager  to  identify  which  LRUs  are  consuming  the  most 
logistics  resources  and  focus  attention  on  those  LRUs. 

^Gebman  et  al.  (1909),  p.  72. 
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The  stages  (rf  maturatian  develt^mient  are  discussed  In  detail  below.  For  this 
discussion,  data  sufficiency  is  assumed  dtfou^K>ut;  a  detailed  discussion  erf  data 
requirements  to  support  maturation  deveU^mtent  is  reserved  for  Section  4. 


Maturation  Development  Through  the  Life  Cycle 

This  subsection  describes  how  maturation  development  would  be  implemertted 
at  each  phase  of  a  weapon  system's  life  cycle,  fmn  concept  exploration  to 
operation  and  support.  In  eadi  phase,  we  focus  on  the  engineering  and  logistics 
activities  because  the  maturation  development  concept  calls  for  tiwse  to  become 
more  integrated.^  Then  we  describe  briefly  tiw  additions  to  each  phase  that  are 
needed  to  implement  a  maturation  development  program.  Table  3.1  summarizes 
these  descriptions. 


Concept  Exploration  and  Definition  Phase 

Purpose.  In  this,  the  initial  weapon  system  development  phase  conceptual 
studies  are  used  to  develop  alternative  solutions  for  design,  production,  logistics, 
and  testing  during  the  devel<qnnent  of  ttie  weapon  system.  The  only  hardware 
are  breadboards  and  prototypes,  typically  focused  on  new  technology 
components. 

Current  Engineering  and  Logistics  Activities.  The  chief  engineering  activities 
are  requirements  atudyses,  functional  analyses,  trade-oti  studies,  preliininary 
allocations,  prelimituuy  dengn  syntiresis,  and  analytic  evaluation  leading  to 
alternative  system  concepts.  Preliminary  fuitctional  allocations  and  error 
budgets  for  subsystems  are  generated  and  configuration  maruigement  baselines 
are  established. 

The  chief  logistics  activities  in  this  period  are  (1)  the  investigation  of  alternative 
support  concepts  to  develop  an  integrated  logistics  support  (ILS)  strategy  and 
(2)  tiie  adoption  of  tttese  concepts  into  file  engineering  ^tem  design  to  mflueiKe 
tile  resulting  product  definition.  Initial  planning  for  the  structure  and  scc^  of 
the  logistics  prcigram  (in  tiie  form  of  ILS)  occurs  in  tills  pericxl.  Currently  tiie 
ioais  of  ILS  is  on  tiie  strategy  and  support  concepts  fat  hard  failures  and  tiieir 
impact  cm  the  support  system. 


^  A  number  of  otter  very  in^iortmt  activttia  (eg.,  oontneting)  also  occur  during  each  phaie 
ttat,  for  brevity's  Mke,  we  do  not  diacuH  here.  Cle^  these  otter  activities  are  also  aStoOed  by  the 
decisions  made  in  logistics  and  engineering. 


Table  3.1 


nan  matoration  data  collection  and  anal3rMa 


Tabic  3.1— conUnued 
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Focus  of  Activities  Under  Maturstion  Development  Undar  maturation 
developmott,  the  purpose  of  this  phase  does  not  diange.  However,  there  is  a 
shift  of  focus  in  the  engineering  and  logistics  activities. 

For  example,  during  the  engineering  studies  for  alternative  functional  solutions, 
more  attention  would  be  given  to  studies  of  tite  costs  and  benefits  of  using  qpen 
architectures  that  would  ease  system  modification  later  in  ti\e  life  cycle.  Open 
architectures  should  also  be  considered  in  the  development  of  preliminary  BIT 
concepts.  Open  BIT  architectures  facilitate  the  later  modification  of  sensing  and 
testing  elements. 

Another  important  added  activity  under  tiw  maturation  development  concept 
during  this  phase  is  explicit  planiung  for  integrated  data  collection  and  analysis 
of  system  malfunctions,  anomalies,  aitd  performance.  The  maturation 
development  concept  includes  and  depends  strongly  on  integrated  data  and 
information  systems.  A  well-developed  management  information  system  linked 
to  an  integrated  RAM  database  is  needed  to  facilitate  efficient  aiui  effective 
resolution  of  RAM  problems.  This  phase  must  include  tire  prelimiiuuy  plaiming 
for  an  integrated  database  to  support  such  anal3rsis,  including  the  capture  of  the 
prototype  phenomena  for  use  in  later  analysis  of  hardware  behavior.^^ 

The  management  mformation  system  must  be  integrated  because,  as  noted 
earlier,  in  a  complex  weapon  system,  RAM  efiects  can  appear  in  tire  system  at 
locatirms  and  echelons  that  are  remote  from  tire  cause.  For  exaix^le,  what 
appears  to  be  a  radar  problem  may  in  fact  be  caused  by  an  iirertial  navigaticm 
system  that  is  providing  erroneous  velocity  information  to  tire  radar.  Because 
fault  indications  in  these  complex  systems  are  not  always  unambiguous,  patterns 
must  be  observed  and  recorded.  This  complexity  requires  data  linkages  across 
the  elements  (hardware,  software,  and  test  eqiaipment)  of  the  weapon  system  aird 
across  time.  These  mairagement  information  systems  must  allow  managers  to 
assess  and  evaluate  across  tire  weapon  and  tire  support  system,  which,  in  turn, 
requires  the  ability  to  lirrk  data  from  1)  operatioirs;  2)  fault  indication  at  tire  end 
item,  including  BIT;  3)  intermediate  test  arrd  repair;  and  4)  depot  test  and  repair 
over  a  period  of  time. 

In  this  irritial  period  of  weapon  system  acqrrisition,  much  work  is  done  tm  critical 
components  such  as  special  integrated  circuits  or  sensor  elements.  At  this  stage, 
the  first  RAM  data  becotrre  available  in  tire  form  of  fiulure  data  on  tire  prototype 
components.  Failure  mode  airalyses  in  later  phases  can  benefit  from  tire  data 


^^The  next  section  addresses  the  cap^ities  and  oomeits  of  die  imegnted  database  in  mote 
detail.  Tt  alffo  rompam  thnr  mpiirrmmtti  nith  thoar  of  ninrnt  datihaiirii 


35 


odlected  and  aiudyxed  in  thtt  eaiiy  fduoe.  The  data  from  engineering  tests  cam 
be  used  to  start  a  descrq>tian  of  the  physics  of  hulure  inodes  that  is  in^iortant  for 
later  analysis.  This  process  can  begin  even  dtough  c<)uipment  exists  only  in 
breadboard  f(»m  and  strftware  exists  only  in  pre-prototype  iotm. 

Demonstration  and  Validation  Phase 

Purpose.  The  purpose  of  diis  frfuue  is  to  identify  and  analyze  ttw  ms^  system 
alternative  configurations  and  to  identify  technkal  and  econ<»nic  risks. 
Demonstraticms  are  conducted  to  decide  between  alternative  designs  for  ttioee 
areas  of  risk  using  developmental  prototypes  of  portions  of  equipments  and 
subsystems.  The  hardware  available  during  this  phase  consists  of  advanced 
development  prototypes. 

Current  Engineering  and  Logistics  Activities.  The  diief  engineering  activities 
are  the  refinement  of  the  studies  done  in  dw  first  jdutte  and  die  extension  of  the 
design  to  configuraticm  synthesis  and  engineering  development  models.  As  test 
and  evaluation  of  significant  pordons  of  equipment  are  conducted,  data  gadwred 
are  used  in  selecting  lower  risk  alternatives  and  further  refining  the  analyses. 

The  chief  logistics  activities  are  the  idottification,  definition,  and  assessment  of 
logistic  implications  of  die  major  system  alternatives  to  influence  die  selection  of 
die  major  system  alternative.  This  selection  includes  an  identification  of 
projected  resources  needed  to  support  die  system.  Among  these  resources  are 
the  types  of  test  equipment,  types  of  ^^ares,  types  and  skills  of  maintenance 
technicians,  and  quantities  of  each.  The  focus  of  the  ILS  in  diis  period  is  on  die 
design  of  the  initial  support  systexxL 

Focus  of  Activities  Under  Maturation  Develr^mcnt  Uruier  maturation 
development,  there  would  be  two  m^or  additions  to  die  engineering  and 
logistics  activities:  (1)  the  capture  of  ancxnaly  data  fiom  prototype  testing  in  an 
integrated  database  and  (2)  planning  for  furdier  refining  die  logistics  support 
system  using  maturation  data  collection  during  LRIP  (described  below). 

The  first  of  these  additional  activities  is  important  because  anomalies  diat  occur 
in  testing  frequendy  seem  to  be  only  curiosities  at  their  first  occurrence.  Ifsudi 
data  are  captured  in  an  integrated  database  that  is  used  later  for  analysis  of 
anomalies  in  operation  and  test  of  die  system,  diey  can  contribute  to 
understanding  die  modes  diat  degrade  performance  (as  opposed  to  hard 
reliability  f ruluies). 
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The  seccmd  additional  activity  is  important  because  tiie  intensive  data  collection 
and  analysis  tiiat  takes  place  during  LRIP  requires  planning  to  ensure  tiuit  tiw 
time  and  resources  tiiat  will  be  needed  are  acquired.  As  noted  earlier, 
maturation  developmort  relies  on  an  aiudysis  of  detailed  data  collected  during 
exteittive  operation  of  the  weaptm  system  in  an  operational  environment.  Time 
and  resources  to  conduct  the  extensive  operatirm  of  the  weapon  system  and 
analyze  the  data  must  be  planned  for.  The  planning  will  omstruct  a  test  program 
that  will  allow  capture  of  detailed  data  on  anomalies  as  they  appear.  Because  tine 
specific  anomalies  cannot  be  predicted  in  advance,  the  test  plans  must  be  goieral 
enough  tu  allow  for  adaptive  testing  as  the  anmnalies  appear.  Such  adaptive 
testing  is  usually  possible  cmly  when  there  is  not  a  detailed  series  of  tests  that 
must  be  passed  in  order  to  proceed  witi\  the  program. 

Engineering  and  Manufacturing  Phase,  Part  1  (Prior  to  LRIPp^ 

Purpose.  The  purpose  of  this  initial  period  of  the  Engineering  and 
Manufacturing  Phase  is  to  identify  and  aiudyze  the  ma^r  alternatives  for  the 
weapon  system,  the  included  equipment,  and  firul  choices  in  the  logistics 
support  structure.  It  is  the  first  time  that  detailed  'T>udgets"  for  R&M  are 
estimated  and  substantial  quantities  of  R&M  data  become  available.  (An  R&M 
"budget"  is  the  detailed  allocation  of  ttw  required  R&M  performance  levels 
across  the  hardware,  software,  and  diagnostic  test  equipment  titat  compose  the 
weapon  system;  in  short,  a  budget  is  an  estimate  of  R&M  performance.)  The 
focus  is  on  the  design  and  test  of  selected  alternatives.  Design  trade-offe  are 
made.  System  tests  are  conducted  using  engineering  prototypes  at  first  and, 
later,  production  prototypes. 

Current  Engineering  and  Logistics  Activities.  Engirteering  activities  include  the 
detailed  design,  syntiiesis,  and  prototyping  of  ttie  weapon  system.  Prototype  test 
results  are  used  to  refine  the  design. 

Logistics  activities  include  actions  to  ntake  ILS  an  integral  part  of  tire  design 
trade-offe  and  to  ccnnplete  ttie  design  of  the  support  system.  Support  items  are 
designed  and  elements  of  the  support  system  are  tested.  The  implementation  of 
ILS  requires  the  development  of  a  detailed  maintenance  support  concept  and 
detailed  estimates  of  the  R&M  of  tite  hardware.  Maintenance  system  design 
entails  the  partitioning  of  the  weapon  system  into  subsystems,  equipments,  and 
LRUs  at  the  end  item.  It  also  defines  the  test  and  diagnostic  structure  covering 


^"Engineering  and  manufaGtiuing’’ is  a  current  DoD  tenn  for  ttie  phase  previouily  refnred  to 
as  hiU-scaie  develo^ent. 


BIT,  diagnostic  procedures,  infeentiediate-level  test  equ^>ment  (if  applicable},  and 
dQ>ot-level  (and/or  productian)  test  equipment 

Focus  of  Activities  Under  Maturation  Development  Under  the  maturation 
development  ccmcept,  three  additional  activities  occur  in  this  initial  period  of  the 
Engineering  and  Manufacturing  Phase. 

The  first  activity  collects  data  during  the  operation  of  the  prototype,  including 
information  about  diagiuMtic  and  anomaly  events.  The  data  must  be  collected  to 
allow  analysis  of  a  sequer\ce  of  activities  over  time.  (See  Section  4  for  a 
discussion  of  the  atralysis  of  sequenced  events.)  Because  it  facilitates  analysis  of 
the  evolving  design,  these  data  must  be  incorporated  into  a  database  that  is 
integrated  across  functions,  echelons,  and  time. 

The  second  activity  uses  titat  data  to  up>date  R&M  budgets.  Some  of  tills 
updating  can  be  tentative  in  tiie  sense  of  evaluating  what  the  implications  would 
be  'f  a  given  anomaly  in  the  factory  engineering  test  appeared  more  fiequently 
than  expected.  Such  R&M  budget  impacts  are  both  resources  and  "flags"  for  the 
evaluaticHi  of  events  in  tiie  intensive  data  collection  period.  Such  budgets  are 
critical  for  tiie  determination  of  required  resources  and  the  control  of  the 
program  to  attain  the  required  R&M  performance.  The  current  RAM  resource 
estimation  process  fiequently  does  not  provide  for  omtinuing  updates  during 
the  life  cycle  of  the  weapon  system  and  does  not  provide  linkages  to  other  data 
and  information  generated  in  nonlogistics  areas  of  the  program.  The  availability 
and  use  of  an  integrated  R&M  data  system  would  allow  for  botii  updating  and 
linking.  Estimates  would  be  derived  from  tiie  integrated  R&M  data  system  and 
could  be  used  to  evaluate  and  prioritize  the  R&M  issues  that  arise  during 
development. 

The  third  activity  updates  and  refines  tiie  plan  for  tiie  intensive  operation  and 
data  collection  and  analysis  that  will  occur  during  LREP.  Planning  in  this  phase 
will  consist  of  five  activities. 

1.  Update  the  more  general  planning  based  on  the  detailed  information 
available  from  productian  prototypes  and  tiie  methods  available  for 
capturing  infonnaticm  in  the  operational  maintenance  and  support  setting. 

2.  Plan  for  the  detailed  capture  of  operational  anomalies  over  all  of  the  edielons 
of  operation  and  support,  including  tiie  relation  of  write-ups,  BIT  indications, 
field  maintenance  procedure  indicatitms,  and  in-factory  or  in-depot  repair 
test  indications. 

3.  Plan  for  adaptive  testing  if  detailed  engineering  data  are  required  for  a 
particular  anomaly,  subsystem,  equipment,  or  module. 
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4.  Plan  the  analytical  effort  to  identify  significant  problems  and  to  evaluate  and 
prioritize  fixes. 

5.  Plan  for  the  development  and  testing  of  modifications  (design  changes  to  the 
weapon  system)  for  the  significant  problems. 


Engineering  and  Manufacturing  Phase,  Part  2  (LRIP) 

Purpose.  As  currently  conceived,  the  purpose  of  this  phase  is  to  verify  the 
production  process  for  the  initial  production  model  and  determine  die 
operatioiud  effectiveness  and  suitability  of  the  weapon  system. 

Current  Engineering  and  Logistics  Activities.  The  engineering  and  logistics 
activities  of  the  Engineering  and  Mrmufacturing  Phase  described  above  continue 
during  LRIP,  making  use  of  the  larger  base  of  data  from  die  low-rate  production 
testing. 

Engineering  activity  centers  on  production  start-up.  This  involves  verifying  the 
planned  production  test  sequence,  dealing  widi  problems  in  the  production 
processes,  and  solving  problems  in  the  production  testing.  Engineering  is  dien 
involved  in  die  production  and  production  testing  of  die  support  equipment 
(including  any  end  item  and  any  intermediate-level  test  equipment). 

Logistics  persormel  are  involved  in  constructing  and  testing  the  support  system 
and  updating  R&M  performance  estimates.  Production  hardware  becomes 
available  to  test  tools,  TMDE,  documentation,  and  procedures. 

Both  engineering  and  logistics  persormel  have  roles  in  updating  the  test  and 
evaluation  master  plan  in  preparation  for  the  operational  test  and  evaluation  diat 
occurs  in  the  LRIP  phase. 

Focus  of  Activities  Under  Maturation  Development  This  phase  of  maturation 
development  concentrates  on  identifying  and  correcting  support  problems. 
Because  reasonable  quantities  of  production  models  are  available  in  a  field 
environment,  extensive  identification,  analysis,  and  evaluation  of  support 
problems  and  potential  corrections  are  possible. 

Under  the  maturation  development  concept,  die  LRIP  phase  adds  important 
activities  that  contribute  to  the  development  of  a  mature  production  model: 


Intensive  operation  of  the  weapon  system  in  the  field  environment,  and  the 
collection  and  analysis  of  detailed  R&M  and  operational  data,  including  the 
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analysis  and  prUmtization  of  mission  effectivenass  and  life<ycle  cost 
impacts. 

•  Design  and  implementation  of  modifications  to  dw  system  as  judged  to  be 
cost-effective. 

•  Verification  of  desired  effects  of  tiie  modificaticms  on  the  system's  R&M 
performance  and  costs. 

From  die  perspective  of  maturation  development  d^  goal  of  the  intensive 
operation  of  the  weapcm  8)rstem  and  data  capture  and  aiudysis  is  not  to  pass  a 
test  requirement  but  to  understand  die  proximate  cause  of  all  anomalies  and 
malfunctions  observed  in  die  field  environment 

Intensive  Operation  of  the  Weapon  System,  Data  Collection,  and  Analysis. 
Under  maturation  development  die  key  activity  during  LRIP  is  an  intensive 
period  of  operation  during  which  data  on  die  R&M  and  operational  performance 
of  fielded  systems  are  collected  and  analyzed.  During  this  period,  die 
configuration  of  die  hardware  and  software  must  be  frozen  to  maintain  a  known 
baseline  for  evaluation.  The  length  of  diis  period  is  a  functitxi  of  the  field  activity 
rates:  generally  speaking,  the  greater  the  activity,  the  more  data  generated,  and 
thus  the  shorter  die  period.  The  data  coUection  occurs  at  each  level  of 
maintenance  in  order  to  permit  a  comprdiensive  evaluation  of  the  system's  R&M 
performance  and  characteristics.  The  detailed  data  collection  permits 
engineering  analyses  of  die  problems,  frequendy  based  on  linking  die  indications 
from  each  echelon  and  from  previous  indications  of  anomalies  captured  in  the 
earlier  acquisition  phases.  (See  T^le  4.1  in  the  next  section  for  specific  data 
elements  that  should  be  collected  during  diis  period.) 

During  die  Engineering  and  Manufacturing  Phase,  a  significant  quantity  of 
deteiled  information  becomes  available  firom  engineering  tests  and  from 
component  production  tests;  this  information  can  be  used  for  detailed  failure 
mode  anal)rsis.  Test  information  needs  to  be  captured  in  die  same  way  that  die 
research-phase  information  on  failure  modes  was  captured.  Similarly  it  must  be 
linked  to  the  engineering  knowledge  of  failure  modes  and  estimates  of  the 
frequency  of  their  occurrence.  The  data  from  factory-  testing  of  compcments  allow 
the  analysis  of  phenomena  that  occur  in  the  factory.  The  results  of  diagnostic 
testing  from  the  end  item  to  factory  testing  of  removed  components  need  to  be 
related  and  tracked  over  time.  This  tracking  involves  resolving  die  results  at 
different  test  edielcms  with  engineering  tests,  including  tests  of  BIT,  manual 
diagnostic  procedures,  factory  test  equipment  and  procedures,  and  component 
test  equipment  and  procedures. 
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The  structure  and  mechanisms  to  capture  and  link  this  informaticm  need  to  be 
developed  and  exercised  early  in  LRIP  in  order  to  ensure  that  at  each  test  level- 
from  initial  indication  of  a  problem  in  the  end  item  through  repair  of  SRUs  in  the 
factory— the  test  results  can  be  related.  This  process  ensures  that  ti\e  mdications 
and  reprur  actioits  at  each  echelon  relate  to  ti\e  same  problem.  This  same 
mformation  and  logic  will  later  be  receded  in  the  field  to  track  particular 
problems  and  their  solutions. 

Because  of  the  need  for  hig^y  detailed,  high-quality  data,  some  elements  of  this 
data  should  be  collected  by  an  esq^ert  team  tiiat  includes  contractor  engineers  and 
techiucians.  Detailed  data  include  a  verbal  description  of  tiie  anomalous  faulty 
behavior  of  the  equipment  and  a  description  of  ti\e  complete  operational  and 
technical  context  of  the  event.  The  team  can  ask  questicms  such  as  the  following: 
"When  in  ttie  operation  did  the  fault  occur?"  "Was  the  fault  repeatable?"  "Does 
it  occur  only  in  certain  situations?"  "Was  tiiere  a  BIT  indication  of  ti\e  fault?" 

The  team  will  also  be  able  to  ask  more  specific  questions  based  on  the  fault 
history  of  the  p2uticular  hardware:  "Did  you  see  X  fiiulty  behavior  under 
Y  conditions?"  The  team  may  also  be  able  to  suggest  specific  data  collection 
activities  if  certain  equipment  behavior  is  observed:  e.g.,  "if  X  fault  occurs, 
please  record  a  certain  computer  memory  location  value." 

Maturation  development  requires  the  capture  of  data  tiiat  mi^t  not  normally  be 
reported  by  an  operator.  Frequently  during  operation,  an  anrnnaly  may  occur 
that  does  not  result  in  a  registered  fault  (ie.,  the  BIT  does  not  recognize  the 
anomaly  as  a  fault  or  the  operator's  assessment  of  the  impact  of  tiie  anomaly  for 
the  next  mission  is  that  maintenance  is  not  required)  but  is  still  an  anomaly  and 
needs  to  be  reported.  Because  the  system  designers  did  not  plan  for  tiiis  type  of 
anomaly,  the  BIT  may  not  capture  tiie  fault.  A  system  as  simple  as  a  tape 
recorder  or  nonvolatile  computer  memory  that  automatically  captures  the 
anomaly,  even  if  only  at  a  major  system  level,  should  record  tiie  environmental 
and  operational  conditions  immediately  preceding,  during,  and  following  the 
anomaly.  In  the  event  such  a  system  fails  to  capture  these  events,  a  debrief  of  the 
operator  should  also  occur  that  includes  indications  of  irregular  performance  that 
would  not  normally  lead  to  a  write-up  or  any  request  for  maintenance.  These 
data  are  needed  to  enable  evaluation  of  patterns  of  anomalies  that  occur  without 
individually  appearing  important  to  the  operator  or  requiring  immediate  actions. 
Such  a  pattern,  if  it  recurs  or  if  it  leads  to  an  unacceptable  anomalous  event,  can 
be  critical  in  understanding  the  physical  and  environmental  phenomena  that 
caused  the  faulty  performance.  The  analyzed  data  can  then  be  used  to  develop 
and  evaluate  solutions  for  the  actual  field  R&M  problems. 


The  participation  of  engineering  perscatnel  in  ttiis  phase  pemuts  the  rapid 
extension  of  planned  testing  to  determine  particular  anomaly  mechanisms, 
operational  interactions,  and  environmental  sensitivity  as  anomalies  appear  in 
operation.  As  opposed  to  "proof"  testing  where  all  of  the  test  points  need  to  be 
defined  before  execution,  intensive  maturation  testing  is  adaptive  in  order  to 
pemut  the  capture  and  definiticm  of  significant  operational  anomalies  aiul 
degradations  in  system  operation  and  support  as  ti\ey  emerge. 

Development  of  System  Modifications.  Beginning  during  the  intensive  data 
collection  and  immediately  following  it  is  the  period  for  developing  and 
prioritizing  modifications  for  the  significant  problems  discovered  during  the  data 
collection.  These  will  be  implemented  tiuough  service/deficiency  reporting  aird 
Engineering  Oumge  Proposals  (ECPs).  Hie  implementation  of  fixes  is  a 
straightforward  process  when  the  vinderl)nng  problem  is  well  understood.  Even 
so,  it  is  critical  to  test  tire  fix  because  tire  degree  of  improvement  can  be  uncertain 
and  it  is  always  possible  to  introduce  a  new  problem  unintentionally. 

Verification  of  Effects  of  Modifications.  The  period  of  developing 
modifications  must  be  followed  by  another  similar  but  less  intensive  operation, 
data  collection,  and  aiudysis  period  after  tire  fixes  determirred  from  the  first 
period's  analysis  have  been  iruide.  The  second  phase  is  used  to  evaluate  the  fixes 
and  assess  their  overall  impact.  The  history  of  the  impact  of  substantial  changes 
on  weapon  system  support  requirements  is  checkered  witir  problems.  Predicting 
the  success  of  a  group  of  changes  based  on  engineering  estimates  of  the  reduction 
in  support  requirements  is  not  an  adequate  means  of  establishing  a  firm  base  for 
proceeding  with  the  fleetwide  changes  in  production  (or  upgrade). 

Production  and  Deployment  Phase 

Purpose.  During  the  production  phase,  full-rate  production  and  fielding  of  a 
highly  capable  weapon  system  ocair.  Hardware  performance  is  verified  to  meet 
specifications.  Program  maruigement  responsibilities  transfer  from  the 
acquisition  to  the  support  communities.  Full-rate  production  equipment  is 
provided  to  field  units  with  support  and  support  eqmpment. 

Current  Engineering  and  Logistics  Activities.  Engineering  and  logistics  identify 
deficiencies  and  work  to  correct  them.  ILS  acquires  all  necessary  support  items. 
For  tile  first  time,  numbers  of  items  imdergo  tiie  producticm  testing  process. 

With  this  experience,  problems  witii  items  not  passing  tests  or  passing  an  initial 
item  test  only  to  fiul  in  a  higher  assembly  test  will  appear.  Early  in  the 
production  process,  tiie  test  may  be  incorrect,  the  production  process  may  be 
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incorrect,  or  ttie  item  may  require  redesign.  All  of  these  situations  will  require 
engineering  changes  to  correct. 

As  productirm  progresses,  more  modifications  occur  in  respoise  to  unacceptable 
yield  rates  of  processes  and  unacceptable  failure  rates  in  higher  assembly  testing. 
As  support  equipment  becomes  available,  its  testing  may  generate  dianges  to 
deal  with  support  equipment  producticm  tests  aiul  interface  problems. 

Focus  of  Activities  Under  Maturation  Development  After  the  intensive 
operation,  data  collectitm,  and  aiudysis  period  and  subsequent  modifications,  the 
data  system  is  used  to  monitor  delivered  R&M  performance,  detect  any  problems 
as  they  arise,  and  evaluate  the  inqjact  of  configuration  changes  as  they  occur. 

The  challenge  to  configuration  control  in  diis  period  is  to  document  both  die 
changes  and  the  testing  of  the  dianges.  Because  die  effects  of  engineoing 
changes  can  interad,  diis  is  a  difficult  task. 

As  the  system  moves  into  high-rate  production,  production  test  data  and 
operational  data  become  available  for  a  much  larger  population  of  items.  Before 
this  phase,  most  of  the  program  effort  focuses  on  design  problems  where  all 
items  of  a  particular  typ>e  exhibit  the  same  characteristics.  Widi  die  larger 
population,  the  data  sjrstem  can  begin  to  track  sudi  problems  as  lemon  LRUs  and 
SRUs.  Identifiable  by  serial  number,  lemons  exhibit  failure  inodes  that  are  not 
the  same  as  those  exhibited  by  other  items  of  identical  design.  Their  failure 
modes  may  be  triggered  by  particular  characteristics  of  the  envinmment  and 
may  not  be  present  in  die  test  bench  environment,  occurring  only  in  die  field 
environment.  Because  diese  problems  do  not  occur  in  the  normal  diagnostic  test 
environment,  they  can  cause  repeated  removals  without  die  associated  test  and 
"repair"  removing  die  cause  of  the  problem.  This  is  one  reason  why  it  is  so 
important  to  link  the  reason  for  an  LRU  or  SRU  removal  to  the  result  of  the  test 
of  that  item  in  repair.  Current  field  expmence  has  shown  diat  die  replacement  of 
some  "out  of  spec"  component  in  maintenance  is  not  equivalent  to  diagnosing 
and  removing  the  problem  that  led  to  die  removal  of  the  component  from  die 
end  item. 

Dealing  with  these  problems  requires  tracking  die  performance  of  individual 
serial-numbered  items  through  Iheir  operational  use.  The  collection  of  all  data 
associated  with  r^ierational  malfunctions,  diagnostic  test  results  at  each  level, 
and  repair  actions  over  time  allow  identification  of  lemon  compcments.  These 
data  can  be  compared  with  die  data  from  die  overall  population  and  patterns  can 
be  established.  When  specific  serial  numbers  ate  identified  widi  anomalous 
patterns,  special  diagnostic  tests  based  on  die  particular  environmental  and 
operational  circumstances  associated  widi  die  anomalies  can  be  devised  to 
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identify  tite  particular  faikire  mode  involved.  After  identification  of  ttie  failure 
mode,  alternatives  can  be  evaluated.  If  alternatives  diat  involve  tailoring 
diagnostic  or  repair  procedures  do  not  appear  cost-effective,  die  component  can 
simply  be  removed  from  service. 

Operation  and  Support  Phase 

Purpose.  Ihis  phase  overlaps  widi  the  previous  phase  of  production  arul 
deployment  because  systems  operate  and  require  support  as  soon  as  they  are 
deployed.  In  dus  phase,  the  operational  cotruxumd  assumes  property 
accoimtability.  Operator-supported  product  improvements  and  modifications 
begin  and  continue  throu^  this  period,  their  number  and  magiutude  depoiding 
cm  the  success  of  production,  continuing  mission  atudysis,  tedmological 
advances,  and  dte  achievement  of  supportability  goals. 

Current  Engineeriiig  and  Logistics  Activities.  Engineering  personrtel  and 
logisticians  monitor  the  delivered  performance  and  develc^  the  user  supported 
changes.  The  formal  reliability  monitormg  at  the  wholesale  level  begins  widi 
fleetwide  failure  data.  Engineering  supports  the  development  of  user-mitiated 
functioiud  upgrades  and  the  reliability  work  in  such  areas  as  fatigue  studies  aiul 
refining  die  requirements  on  time-change-out  components.  Maintainability  is 
primarily  addressed  througjh  reports  from  die  field  regarding  deficiencies  in 
manuals  and  procedures. 

Focus  of  Activities  Under  Maturation  Development  In  addition  to  monitoring 
and  identifying  lemon  LRUs,  diis  phase  of  maturation  is  used  to  continuously 
monitor  the  R&M  performance  as  configuration  changes  are  made.  Frequendy, 
engineering  changes  made  for  functional  improvement  or  safety  concerns  result 
in  unexpected  RficM  problems.  Mcxlifications  also  offer  die  possibility  of 
itKorporating  R&M  changes  that  would  not  be  cost-e&ective  to  incorporate 
separately.  There  is  also  a  continuing  need  to  monitor  and  evaluate  die 
maintenance  and  support  system  as  the  fleet  grows.  Some  problems  diat  will  be 
important  with  large  numbers  of  ecjuqnnent  or  appear  as  a  consecpience  of  aging 
do  not  occur  early  in  die  life  of  a  weapem  systeiru  Thus,  the  data  system  is 
needed  throughout  die  fife  of  the  weapon  system,  even  after  productiem  is 
finished  and  the  system  is  in  full  operation.  Some  special  data  collection  and 
analysis  efforts  may  be  needed  dirough  the  course  of  die  system's  fielding; 
however,  as  a  rule  it  is  not  necessary  to  continue  data  collection  and  analysis  at 
die  level  of  intensity  applied  during  LRIP. 
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Summary 

The  maturation  development  concept  starts  with  dw  recognition  of  the 
extraordinary  technical  complexity  of  new,  high-tech  weapon  systems  and  the 
understanding  that  when  such  systems  are  newly  fielded  ttiey  will  not 
immediately  deliver  dieir  full  designed  performance.  It  goes  on  from  there  to 
acknowledge  that  high-tech  electronics  iteed  to  nuture,  and  that  maturatian  can 
be  numaged  by  weapon  system  sustainment  maiuigers  throughout  the  life  cycle 
of  d\e  weapon  system.  This  contrasts  %vith  the  traditianal  view  of  mechaiucal 
systems  that  R&M  characteristics  are  an  output  of  the  initial  design  phase  and 
dtat  once  design  is  over  the  job  is  basically  done.  This  recogiution  of  dte  special 
character  of  high-tedi  ccni^Mjnents  reqtiires  planning  for  an  intense  maturation 
period  early  in  the  fielding  of  a  new  weapon  system  and  plaxming  for  an 
integrated  R&M  data  system  to  support  maturatian  of  die  R&M  diaracteristics  of 
the  weapon  system  over  the  Ufe  of  the  system.  The  benefits  of  sudi  an  effort  are 
both  improved  weapon  system  availability  and  savings  in  total  life  cycle  costs. 

The  concept  also  recognizes  die  need  for  system  assessment  and  evaluation  of 
causes  radier  duui  of  effects  only.  In  a  complex  weapon  system  (induding  the 
support  system),  R&M  effects  can  app«a  in  die  system  at  locations  and  edielons 
that  are  remote  from  the  cause.  Indications  may  be  ambiguous  in  isolation,  so 
patterns  must  be  observed  and  analyzed.  To  assess  R&M  performance  of  die 
weapon  s)rstem  accurately  requires  the  ability  to  link  data  from  operations  and 
from  all  echelons  of  repair  over  a  period  of  time.  Dealing  with  diis  complexity 
requires  linkages  across  die  elements  (hardware,  software,  and  test  equipment) 
of  the  weapon  system  and  across  time  diat  have  not  been  requirements  of 
traditional  R&M  data  systems. 

The  main  difierence  in  approach  between  diis  new  concept  and  the  way  high- 
tech  R&M  is  currently  managed  involves  die  analysis  of  data  to  reveal  pattems 
that  indicate  anomalies  and  their  causes.  To  do  that,  it  is  essential  that  the 
weapon  system  sustainment  manager  track  sequences  of  indications  and  actions 
over  time  rather  dtan  only  counting  events  at  an  edielon  (removals  at  die 
weapon  system,  for  example).  An  intensive  maturation  development  period 
immediately  following  fielding  of  die  weapon  system  and  an  integrated  R&M 
data  system  form  the  basis  for  analysis. 


4.  Data  Requirements  to  Support 
Maturation  Development 


The  second  section  showed  ttiat,  witti  higlHech  electronic  coin|>onenls,  actual 
RfcM  performance  can  fall  £sr  short  designed  RIcM  perfonnance.  HiglHedi 
components  are  extraordinarity  difficult  to  mamtain.  As  iK>ted,  ttiey  suffer  from 
deficiencies  in  fault  detection,  fiiult  isolation,  and  fault  removals.  AnRfcM 
perfonnanoe  shmtfall  has  serious  consequences  both  for  support  costs  and  for 
operational  capability.  Ihe  overall  objective  of  maturation  devdopmertt  is  to 
provide  the  designed  RAM  performance  at  lower  life  cycle  costs.  It  pursues  ttiis 
objective  duoug^  two  approaches.  One  is  to  iderttify  and  correct  RlcM  design 
defidendes-parficulariy  maintamability  defidendes-as  early  as  possible  in  the 
develofnncnt  cycle  of  the  weqxxt  system.  The  other  is  to  identify  and  remove 
lemons  as  they  develop.  The  ability  to  capture  artd  link  data  from  diffaent  time 
periods,  from  different  systems,  and  from  different  sources  is  needed  to 
implement  die  maturation  devdopment  process  diroug^  eadi  |diase  of  the 
weapon  system  Hfecyde. 

This  section  first  describes  qiedfic  capabilities  that  are  required  for  the 
implementation  of  maturadon  development  Ihese  capabilities  are  based  iqxxi 
die  availability  and  use  <rf  a  database  diat  is  integrated  over  time,  across 
functions,  and  across  echdons.  The  required  capabilities  are  contrasted  briefly 
widi  die  capabilities  of  two  of  die  best  current  RfcM  data  systems  eirq>loyed  by 
the  Services.  The  section  then  oudtnes  dte  qpedfic  data  elements  required  to 
support  maturation  devdofmient  and  doses  widi  suggestions  on  reducing  the 
costs  and  burden  of  collecting  and  integrating  data. 

Capabilities  Required  to  Implement  Maturation 
Development 

The  maturation  devek^mient  concept  highlights  die  need  for  a  number  of 
caprdiilities  in  order  to  identify  and  diminate  maintainability  design  problems 
and  lemons.  These  include  the  following  capabilities; 


•  Assess  IUeM  performance  accurately. 

•  Identify  patterns  and  sources  of  RScM  anomalies. 

•  Evaluate  effects  of  potential  fixes  on  weqion  system  availability. 
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•  Diagnoae  ambiguous  faults. 

•  Identify  lemons. 

All  of  dtese  capabilities  require  ttie  develojpn  ent  and  use  of  an  mtegrated 
database  that  tedUtates  die  analysis  of  sequences  of  events.  Sudt  a  database 
would  also  supp<Ht  other  inqxntant  capdnlities,  sudi  as  the  fcdlowing: 

•  Provide  management  reports. 

•  Schedule  periodic  xiuuntenance  ^fidently. 

•  ^pport  manpower  estimation  and  spares  requirements  determination. 

Each  of  tiiese  capabilities  is  described  below,  followed  by  a  discussion  of  die 
integrated  database  itself. 

Assess  R&M  Performance  Acairately 

In  order  to  nnanage  die  R8cM  performance  of  a  weapon  system  edectively,  it  is 
essential  to  have  the  capability  to  assess  R&M  performance  accurately.  Ihis 
entails  being  able  to  measure  the  true  rdiabOhy,  the  true  maintainability,  and  the 
true  logistics  supportabili^  of  die  system.  Once  these  ate  known,  die  weiqion 
system  availability  can  be  calculated  for  a  specified  missifln.  Measuring  die  true 
reliability  and  maintainability  requires  detecting  and  reporting  all  indications  of 
R&M  anomalies,  including  fault  observations  by  the  weiq>on  system  operators. 
Because  rrudntainability  concerns  die  existence  of  faults  that  repeat  after  a 
maintenance  action,  anomalies  must  be  identifiable  when  possible  by  component 
serial  number.  Measuring  logistics  supportabili^  requires  data  on 
administrative  and  transit  times. 

Identify  Patterns  and  Sources  of  R&M  Anomalies 

Because  hi^tech  components  frequendy  display  anomalies  that  are  ambiguous, 
it  is  essential  to  have  the  capability  to  identify  patterns  of  R&M  anomalies  in 
order  to  improve  die  diagnosis  of  problems.  Establishing  patterns  requires  being 
able  to  link  data  from  different  times,  functions,  and  edidons.  In  addition,  it 
should  be  possible  to  link  R&M  performance  data  widi  data  regarding  die 
operating  environment  and  the  missian.  Some  anomalies  may  manifest 
themselves  only  under  special  operational  or  environmental  circumstances  (e.g., 
extremely  cold  weather).  A  full  picture  of  die  performance  history  of  higfy-tech 
crenpreients  supports  the  development  and  testing  of  hypodieses  regarding  die 
source  of  anomalies. 
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Data  should  be  integrated  over  tiane,  begiiuung  early  in  ttie  develo|nnent  cyde. 
Information  obtained  in  the  researdt  and  development  phases  can  be  of  use  in 
later  time  periods  for  diagnosis  arul  analysis  if  it  is  captured  in  forms  drat  are 
appropriate.  Plans  for  later  use  of  the  data  must  recognize  that  diderent 
fururtional  systems  will  be  used  in  later  time  periods.  For  example,  the 
information  on  failures  of  prototypes  is  currently  captured  in  manufacturing  data 
systems  for  use  by  engineers  in  the  identification  arul  resolution  of 
manufacturing  problems.  This  same  mfoimation  may  be  useful  later  in 
developing  an  understanding  of  the  frequency  and  occunerKe  of  failure  modes 
in  later  periods.  Engineering  may  correctly  conclude  tiiat  a  particular  failure 
mode  is  an  isolated  mstaru%  in  manufacturing,  but  knowledge  of  its  occuneiKe  is 
needed  later  to  understand  how  it  could  relate  to  failures  if  it  turns  out  tiuit  field 
conditions  trigger  this  failure  mode. 

Once  a  system  is  fielded,  mainterumce  data  should  be  integrated  across  edwlons. 
For  example,  ttie  database  should  permit  a  nuuntamer  at  an  intermediate  level  to 
learn  whether  fault  detection  and  fatUt  isolation  results  from  his  or  her  tests 
confirmed  the  symptom  observed  at  tiie  unit  level  tiuit  led  to  tite  removal. 
Likewise,  the  database  should  also  permit  a  depot-level  rruuntainer  to  determine 
whether  depot  fault  detection  arul  isolation  results  were  consistoit  witi\  tiw 
intermediate  results.  Such  confinruitums  are  needed  to  link  corrective  actions  to 
observed  faults. 

Evaluate  Effects  of  Potential  Fixes  on  Weapon  System  Availability 

Once  the  source  of  an  R&M  problem  has  been  identified,  alternative  potential 
fixes  can  be  developed  and  evaluated  in  terms  of  their  relative  costs  and  effects 
<si  the  availability  of  truly  FMC  weapon  systems.  An  integrated  database  can 
support  the  assessment  of  potential  fixes  by  providing  baseline  perforrrumce 
data— data  that  can  be  used  to  help  estirruite  tiie  efiects  of  alterrutive  fixes. 

Once  piotential  fixes  have  been  evaluated,  ECPs  can  be  developed  arul 
implemented  when  foimd  to  be  affordable  and  cost-efiective  over  the  remaining 
fielded  life  of  the  weapon  s)rstem. 

Diagnose  Ambiguous  Faults 

As  noted  in  previous  sections,  high-tedt  compcments  fiequently  present 
anomalies  that  are  ambiguous.  One  of  the  strategies  of  maturation  development 
is  to  greatly  reduce  such  ambiguity  by  arudysis  of  more  data  titan  are  currently 
available.  In  addition  to  using  tite  database  to  provide  tite  capability  to  correct 
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hardware/software  defickndes  prior  to  full>rate  production,  it  could  be  used 
through  the  life  of  the  system  as  a  diagnostic  tool.  Ideally,  a  successfully  run 
modification  program  would  elimiruite  all  ambiguity  in  favilt  detection  and  fault 
removal;  urdortunately,  it  is  unrealistic  to  expect  that  degree  of  success.  In  fact,  a 
soluticm  to  the  ambiguity  problem  may  be  to  use  an  integi .  ;ed  database  and 
improved  maintenance  procedures  to  isolate  faults. 

Identify  Lemons 

The  source  of  an  R&M  problem  may  lie  eidter  in  the  weapon  system  design 
(broadly  construed  to  include  the  design  of  a  component,  test  equipment, 
operating  or  maintenance  procedures,  and  training)  or  in  the  anomalous 
behavior  of  an  individual  component— a  lemon 

The  key  to  identifying  lemons  is  to  collect  and  atudyze  operational  and 
maintenance  data  by  component  serial  number.  If  diis  is  done,  eadt  individual 
component's  contribution  to  operatiorul  capability  and  to  the  support  burden 
can  be  calculated,  and  those  components  whose  RAcM  perfomumce  is  chronically 
poor  can  be  identified  and  corrected,  perhaps  even  removed  fiom  service.  Once 
identified,  lemons  can  be  subjected  to  a  special  fault  isolation  program  and  then 
fixed  or  disposed  of.  A  special  fault  isolation  program  is  needed  because 
components  with  such  hig^  removal  rates  generally  have  problems  that  caxmot 
be  detected  or  isolated  using  standard  diagnostic  testing  schemes.  They  require 
engineering  tests  based  on  their  particular  s3mnptoms. 

Provide  Management  Reports 

The  reporting  structure  provided  by  the  integrated  data  system  can  directly 
support  managers.  Such  a  structure  must  support  two  difierent  kinds  of  reports: 

1.  Those  that  occur  at  regular  intervals  and  give  a  picture  for  numagement  and 
control  of  the  R&M  status  and  prioritized  R&M  problems  of  fire  weapon 
system. 

2.  Those  reports,  available  on  request,  duit  allow  analysis  and  investigation  of 
the  linkages,  patterns,  and  probable  causes  of  R&M  performance  shortfalls 
that  reqviire  action. 

This  is  in  sharp  contrast  to  most  R&M  data  systems  today  that  require  a  separate 
study  into  the  possible  causes  of  fire  prr^lem  as  it  surfaces.  An  engineering 
study  of  the  situation  will  still  be  required  to  develop  and  evaluate  alternatives. 
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but  such  study  can  begin  wid\  a  clearer  jncture  ol  die  area  for  investigMion  and 
possible  areas  that  are  associated  widi  die  proUem. 


Schedule  Periodic  Mamtenance  Efficiently 

The  integrated  R&M  data  system  can  be  used  to  evaluate  potential  dumges  in 
sdieduled  maintenance  for  improved  R&M  perfocmance  as  wdl  as  for  potential 
cost  avoidances.  For  some  components-such  as  time  between  overhaul  (TBO) 
items-diere  are  periodic  maintenance  actuxis  diat  interact  widi  modification 
work  orders  in  the  fielded  fleet.  While  the  safety  aqiects  of  diese  are  closely 
tracked  today,  diere  is  little  linkage  to  other  efleds.  sudi  as  availability  of  the 
weapon  system  aiul  impacts  on  sustainatnlity.  To  maintain  diis  linkage, 
reliability  and  maintamability  budgets  need  to  be  maintained  and  updated  to 
allow  tmderstandmg  of  die  related  inqxxtance  of  problems  and  dieir  impad  on 
the  weapon  system  cost  and  R&M  performance. 


Support  Manpower  Estimation  and  Spares  Requirements 
Determination 

Current  data  systems  and  spare/replacement  parts  requirement  determination 
processes  assume  diat  the  R&M  problem  is  a  one-time  problem:  That  is,  optimal 
repair  level  analyses  are  conducted  once.  These  set  the  logistics  parameters  diat 
drive  the  spares  requirements  determination  processes.  Once  these  parameters 
are  set,  item  managers  use  historical  data  to  determine  buys,  repairs,  and 
distributions.  An  integrated  database  can  help  improve  sudi  decisions.  R&M 
data  are  critical  in  determining  die  logistics  resources  required  (e.g.,  ^>ares)  and 
controlling  the  application  of  those  resources. 

Capabilities  of  the  Best  Existing  Databases 

Amcmg  the  data  systems  for  current  fielded  systems,  there  are  no  exarrq>les  of 
fully  integrated  R&M  databases  of  die  sort  needed  to  support  maturation 
development  However,  a  few  existing  data  systems  include  significant  elements 
of  sudi  a  database.  Two  examples  are  the  Army's  data  system  for  the 
TADS/PNVS  system  for  die  Apadie  and  die  Air  Force's  TTCARRS  (Tactical 
Interim  CAMS  (Core  Automated  Maintenance  System]  and  REMIS  [Reliability 
and  Maintainability  Information  System]  Reporting  System)  data  system  for  die 
F-16  and  F-15E  filter  aircraft 


50 


These  are  both  ccmtractor-operated  data  systems  that  generate  infonnation  f<x 
project  and  logistics  management  for  these  weapon  s3^stems.  The  TICARRS  data 
are  gathered  using  remote  termmals  atui  the  Air  Fmce  base  data  systenns.  The 
TADS/PNVS  data  are  gadiered  using  remote  computers.  Both  systems  cover 
maintenance  at  die  end  item,  intermediate  maintenance,  and  depot  maintenance. 
Serial  number  data  for  LRUs  and  end  items  (e.g.,  tank  or  helio^ter)  are  captured 
so  that  the  data  can  be  used  to  follow  maintenance  history  at  die  end  item,  die 
configuraticm  and  modification  status  of  the  end  item,  subsequent  maintenance 
events  on  removed  components  (serialized),  and  summary  of  events  at  each  level 
of  maintenance.  This  coverage  across  the  fleet,  across  maintenance  echelons,  and 
linked  to  the  initiating  event  is  an  important  element  of  die  integrated  data 
system  discussed  above. 

In  other  areas  these  examples  of  the  best  current  practice  are  less  complete.  They 
do  not  capture  anomalies  diat  are  observed  but  do  not  require  maintenance. 

They  capture  only  limited  information  on  the  (^leratiim  of  die  equipment; 
operating  hours  between  maintenance  events  for  die  Army  system  and  operating 
hours  for  each  mission,  and  number  of  missions  for  die  Air  Force.  Data  on  test 
results  are  limited  to  entries  in  text  fields  and  are  fiequendy  not  entered.  The  use 
of  test  result  data  is  limited  by  die  abseice  of  dictionaries  to  relate  the  test  results 
at  different  test  levels. 


Data  Elements  Needed  to  Support  Maturation 
Development 

To  this  point  we  have  described  the  capabilities  of  an  integrated  database  to 
support  maturation  development  This  subsection  describes  and  illustrates  die 
specific  data  elements  that  would  be  collected  in  the  integrated  database 
associated  with  maturation  development  The  data  elements  are  summarized  in 
Table  4.1.  They  fall  into  five  ma^or  categories: 

•  Data  associated  widi  the  weapon  system  at  die  unit  level. 

•  Data  associated  with  intermediate  repair. 

•  Data  associated  with  depot  or  contractor  repair. 

•  Data  associated  with  TMDE. 

•  Data  associated  with  administrative  and  transportation  processes. 

The  discussion  focuses  on  data  to  be  collected  after  the  fielding  of  die  system. 
The  integrated  database  would  also  carry  forward  data  fiom  earlier  phases 
(engineering  and  ILS  development). 


Table  4.1 


Table  41— continued 
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Data  Associated  with  the  Weapon  System  at  the  Unit  Level 

The  first  set  of  elements  is  associated  directly  wittt  die  weapon  system 
compcments  at  the  unit  level  (frequently  a  part  of  organizational  maintenance). 
For  unscheduled  maintenance,  these  include  (1)  symptoms  in  oporation,  such  as 
comments,  date  and  time,  BIT  indications,  and  the  point  in  die  operational 
sequence  when  the  anomaly  occurred,  and  (2)  maintenance  test  results, 
maintenance  performed,  items  replaced,  serial  numbers  and  operational  time  for 
the  installed  and  removed  items,  and  identity  of  maintainers.^  The  data  system 
should  also  capture  operational  anomalies  (ie.,  those  reported  by  operators) 
when  no  maintenance  was  requested  or  performed  so  that  recurring  anomalies 
can  be  identified.  Linked  to  this  data  should  be  information  about  scheduled 
maintenance-e.g.,  date  and  time,  maintenance  performed,  identity  of 
maintainers,  and  items  replaced  with  above  detail.  Also  linked  to  these  elements 
should  be  data  on  the  test  equipment  (if  any)  used  at  the  unit  level-e.g.,  usage, 
operating  time,  maintenance  actions,  and  anomalies  witii  test  equipment 

Data  Associated  with  Intermediate  Repair 

The  second  set  of  data  elements  for  R&M  is  associated  witii  intermediate  repair. 
Data  captured  here  are  on  items  removed  from  tiie  end  items  and  on  components 
used  in  the  repairs.  The  data  collected  are  date  and  time  in  work,  the  test  time  on 
test  equipment,  test  indications  of  faults,  tiie  time  to  repair,  maintenance  done 
and  parts  used,  identity  of  maintainers,  awaiting  parts  time,  sell-off  test  time,^ 
and  the  date  and  time  the  repaired  part  is  aveulable.  These  elements  need  to  be 
linked  to  the  unit-level  data  listed  above. 

Data  Associated  with  Depot  or  Contractor  Repair 

The  third  set  of  data  elements  comprises  tiiose  that  are  available  at  the  contractor 
or  depot.  These  include  repair  data  such  as  date  and  time  in  work,  test  results, 
components  replaced  (including  serial  numbers  where  applicable),  maintenance 
or  repair  performed,  identity  of  mainbuners,  awaiting  parts  time,  and  date  and 
time  available.  Overhaul  date  and  associated  removals  should  also  be  included. 


recognize  that  data  on  individual  maintainer  perfoimanoe  are  sensitive.  Nevertheless,  we 
advocate  its  collection  because  it  may  be  useful  in  identifying  maintainers  who  are  especially 
successful  in  diagnosis  or  repair.  The  expertise  of  such  indi^uals  may  be  studied  a^  important 
lessons  learned  can  be  incorporated  into  training  or  doctrine. 

^^The  time  it  takes  to  confirm  that  a  repaired  component  is  in  feet  repaired  (Le.,  ready  to  be 
"sold  ofT  again). 
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Data  Associated  vnth  JMDE  at  All  Levels 

The  fourth  set  of  data  focuses  on  diagxK)stic  test  equipment  at  all  levels.  TMDE 
capacity  has  an  important  impact  on  weapcm  system  siistaiiuibility.  These  data 
should  include  operating  time,  confidence  test  time,^  diagn(»tic  test  time  while 
troubleshooting  the  test  equipment  itself,  repair  time  for  the  test  equipment, 
parts  used  with  serial  numbers  where  applicable,  and  test  eqtiipment  adapters, 
including  software  and  test  equipment  configuration. 

Data  Associated  xmth  Administrative  and  Transportation 
Processes 

The  fifth  set  of  data  elements  focuses  on  an  integral  part  of  the  maintenaiKe 
process,  the  administrative  and  transportaticm  processes  used  in  the  support 
system.  Data  elements  include  tuin>in  date  and  time  of  failed  items,  shipping 
date  and  time,  nodal  times  in  die  transportation  system,  movement  times  in  the 
transportation  system,  arrival  date  and  time(s)  in  the  supply  system,  date  and 
time  to  maintenance  in  the  supply  system,  and  date  and  time  from  maintenance. 
These  data  elements  should  cover  bodi  die  order  and  ship  segments  and  the 
retrograde  segments. 


Lessening  the  Cost  and  Burden  of  Data  Collection 

Extensive  data  must  be  collected  to  capture  die  information  required  to  identify 
and  numage  R&M  problems  of  hig^<tech,  high-cost  LRUs.  Data  about  high-tech 
subsystems  need  to  be  captured  in  detail;  existing  data  about  weapon  system 
operation  (in  the  form  of  log  book  data)  need  to  be  integrated  with  other  data  so 
that  they  can  be  accessed  and  used.  Because  data  collection  and  analysis  is 
expensive,  collecting  detailed  data  on  every  item  of  a  weapon  system  may  not  be 
economically  feasible.^  A  comprehensive  integrated  data  system  may  be 
justifiable  only  for  selected  high-tech  subsystems  of  important  weapon  systems. 
Research  is  needed  to  establish  the  requirements  for  detailed  data  collection  and 
to  determine  methods  of  collection  and  integration  that  are  affordable. 


time  it  takes  to  test  the  IMDE  itself  to  confirm  that  it  can  be  used  with  confidence  to  test 
a  suspected  bad  component. 

^^Some  of  the  expense  can  be  spread  across  weapon  systems.  For  example,  die  infcmnation 
systems  to  support  lemon  detection,  while  costly,  co\^  be  designed  so  that  t^  need  be 
implemented  o^y  once  for  die  Army  maintenance  system.  Afin  providii^  the  capability  to  collect 
and  analyze  such  information,  that  capability  can  be  used  virtually  fiee  for  all  later  acquisitions. 


Technology  can  be  used  to  reduce  both  cost  of  collecting  data  aiul  dte  burden 
on  the  iruuntenance  and  operatiais  persortnel.  Ideally,  if  an  item  is  serial 
numbered  it  would  be  tracked  in  an  integrated  database.  However,  as  a  practical 
matter,  ttie  lack  of  automated  data  collection  prohibits  collecting  data  on  all  serial 
numbered  items.  Many  of  the  data  required  for  aiudysis  could  easily  be  made 
available  in  today's  high-tech  s)rstems  but  are  not  being  captured.  Simple 
technologies,  like  recording  devices,  could  be  used.  Just  now  being  developed 
are  "snuurt  chips"  that  can  be  built  into  circuit  boards  and  perform  Ic^cal 
diagnoses  of  the  board  to  which  it  is  attached  and  store  and  transmit  that  data  to 
the  next  higher  assembly.  Such  a  "bottoms  up"  design  of  LRUs  and  subsystems 
would  ensure  the  capture  of  adequate  information  for  all  levels  of  maintenance 
and  engineering.  This  process  would  allow  technicians  to  capture  the 
mformation  by  direct  access  to  the  diip,  SRU,  or  LRU  without  need  for  additional 
recording  devices. 

Physical  data  capture  mechanisms  should  be  designed  to  feed  automatically  mto 
the  integrated  R&M  management  system  in  order  to  reduce  the  burden  on  ttte 
operators  and  maintainers  and  provide  the  necessary  information  for  R&M  fixes. 
The  data  systems  must  be  designed  to  capture  fire  physical  structure  of  the  data 
and  fiieir  logical  linkages.  The  physical  structure  must  include  data  from  all 
echelons,  from  occurrence  of  the  fault  fiuough  final  removal  of  the  fault.  The 
logical  linkages  must  be  present  in  order  to  ensure  that  each  removal  and  repair 
action  is  logically  connected  to  a  reported  fault 
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5.  Implementation  of  Maturation 
Development 


In  this  section  we  discuss  briefly  some  of  ttw  actions  fluit  can  be  taken  by  flie 
Army  now  to  improve  the  availability  of  FMC  weapon  systems.  We  start  by 
reviewing  steps  proposed  in  flus  report  to  in^nove  the  availability  of  weapon 
systems  that  are  in  the  early  stages  of  researd\  and  development  We  follow  fluit 
wifli  a  discusskm  of  what  steps  can  be  taken  now  to  improve  flie  availability  of 
fielded  high-tech  weapon  systems.  We  note  the  necessity  of  taking  early  action 
to  manage  high-tech  R&M  performance.  Acting  early  makes  it  possible  to  justify 
the  costs  of  flie  maturation  develr^mient  process-including  potential  engineeting 
modifications-in  terms  of  direct  support  savings  over  flie  life  cycle  of  the 
weapon  system.  We  conclude  by  addresang  varknis  obstacles-teduiical, 
environmental,  and  organizatumal— to  implementing  maturation  development 


Applying  Maturation  Development  to  New  Weapon 
Systems 

It  is  important  to  emphasize  fliat  hig^tedi  weapon  systems  will  have  RfcM 
problems  when  first  fielded,  problems  that  cannot  be  completely  obviated  even 
by  the  highest-quality  design  and  production.  Key  decisionmakers  attuned  to 
this  fact  will  seek  to  determine  whether  flie  inevitable  RfcM  problems  are 
crmsequential  and  if  they  can  be  solved. 

We  have  proposed  a  concept,  maturation  development,  for  identifying  and  fixing 
theR&M  problems  associated  with  new  hig^vtedi  weapon  systems.  Thisconcept 
should  enable  program  managers  to  reduce  the  risk  of  acquiring  weapon  s3rsteins 
fliat  are  not  affordable  or  sustainable  over  the  life  of  flie  systeou  Iheconcept 
proposes  that  the  program  manager  design  the  hardware  frcnn  flie  beginning  so 
that  it  can  be  easily  modified  to  correct  deficiencies  found  in  later  stages  of 
development.  To  track  flie  performance  of  fliat  design,  an  integrated  data  ^rstem 
is  established  to  capture  relevant  data  and  link  flie  des^  to  past  and  present 
data  and  data  yet  to  be  collected.  A  key  step  of  fliis  concept  is  an  intensive  data 
collection  phase  that  occurs  during  LRIP  and  is  used  to  identify  and  anafyze 
deficiencies.  The  data  collected  from  this  and  previous  phases  will  permit  a 
comprehensive  analysis  and  identificaticm  of  alternative  solutions  prior  to  full- 
rate  production.  Improvements  to  fault  detection  and  isdation  s3rstems  should 
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be  easily  aocomnodated  because  of  ttie  design  ardutectuie  used  in  the  eaiher 
phases  of  the  program.  Cmtinued  monitoring  of  modifications  and  fielded 
weapon  systems  should  be  automatic  and  at  far  less  of  a  burden  to  operators  and 
maintainers  toan  current  systems. 

Current  legislation  and  DoD  direction  establishes  a  critical  role  for  die  Assistant 
Secretary  of  toe  Army  for  Research,  Development,  and  Acquisition  (SARDA)  in 
this  process.  As  toe  Service  acquisition  executive,  he  is  responsible  for 
establishment  of  acquisition  policy  and  review  of  programs  for  advancement 
through  the  acquisition  process.  In  Section  2  we  presented  much  evidence  from 
studies  of  hi^-tech  systems  indicating  a  ctsisiderable  amount  of  risk  associated 
with  the  R&M  of  high-tech  weapon  systems.  As  a  means  of  reducing  program 
risk,  the  SARDA  could  establish  a  policy  requiring  that  program  managers 
identify,  in  their  periodic  revieivs,  any  process  or  system  they  have  to  manage 
toese  risks.  (Currently,  managers  have  little  incentive  to  highlight  RAM  risks 
after  the  selection  of  system  alternatives  toat  occurs  in  toe  demonstration/ 
validation  phase,  when  technical  risks  and  economic  uncertainties  are  identified.) 
The  program  manager  could  reduce  the  RAM  risk  by  using  an  integrated  data 
s)rstem  in  conjunction  with  an  intensive  data  collection  period  such  as  toat 
described  in  Section  3. 

A  second  policy  that  could  be  implemented  relates  to  the  criteria  used  to  evaluate 
a  weapon  system's  readiness  to  move  to  hiU-rate  production.  Currently, 
measures  such  as  mean  time  to  repair  ate  used  to  assess  a  system's 
maintainability.^  A  measure  such  as  "fault  removal  efficiency"  toat  captures  toe 
effectiveness  of  fault  detection  and  reporting  as  well  as  fault  isolation  is  more 
meaningful.  This  measure  could  be  reported  at  toe  production  decision 
milestone.  Means  to  measure  tois  value  would  need  to  be  incorporated  into  the 
testing  program. 

Given  toe  current  acquiation  envirrmment,  toe  Army  can  expect  to  develop  and 
field  few  major  systems  in  the  1990s.  Nevertoeless,  if  policies  such  as  toese  were 
implemented  now,  the  Army  would  be  in  a  position  to  apply  maturation 
development  to  toose  new  systems  that  it  does  have  some  prospect  of  acquiring, 
such  as  toe  Comanche  and  the  Longbow. 


^White  of  some  value,  tb«aeiMMutes  have  many  liinitations.  Actual  tune  to  repair  an  UtU  is 
determined  by  factors  Budi  as  spam  availability  and  test  equipment  availability.  A  full  discussion  of 
current  maintainability  measures  is  provided  in  Gebman  et  aL  (1989). 
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Applying  Maturation  Development  to  Fielded  Weapon 
Systems 

Maturation  development  as  described  in  Section  3  of  this  report  addresses  new 
systems  that  are  in  leseardi  and  devek^nnent;  yet  the  Army  is  more  commonly 
faced  with  the  upgrading  of  weapon  systems  that  are  already  fielded.  Some  of 
these  modifications  are  large  and  coo^lex  enough  to  juriify  maturation. 
Fortunately,  maturation  development  can  be  adapted  for  application  to  fielded 
systems.  The  maturation  development  concept  relies  on  data  collection  and 
integraticn,  followed  by  pnAlem  identification  and  analysis,  which  leads  to 
alternative  development  and  evaluation.  These  activities  can  apply  to  improvii^ 
the  R&M  perfomumce  of  fielded  systems  as  well  as  new  ones. 

As  the  DoD  enters  a  phase  of  reduced  spending  on  new  acquisitions  and  greater 
interest  in  upgrading  existing  systems  to  meet  new  requirements,  an  opportunity 
exists  to  mature  dte  design  of  these  Belded  systems.  The  Army  could  start  now 
by  taking  actions  such  as  ttie  fbllowmg: 

•  Collect  data  identified  in  Section  4  on  the  high-tech  systems  of  deployed 
weapon  systems  (sudi  as  the  fire  ctmtrol  subsystem  of  the  Apache,  tfw  mast 
mounted  sight  on  the  Kiowa,  the  electrical  section  of  die  lautKher-loader 
module  on  die  Multiple  Launch  Rocket  System  [MLRS],  and  die  fire  control 
system  on  the  MlAl  tank). 

•  Analyze  die  data  to  determine  die  true  fiiult  removal  efficiency  of  these 
systems  and  to  identify  die  most  troublescHne  R&M  deficiencies.  Sudian 
analysis  could  be  used  to  identify  fiiult  detection,  fault  reporting,  and  fault 
isolatitm  problems  as  well  as  lemons. 

•  Develop  engineeiing  fixes  to  these  problems  and  mature  diem  in  the 
development  process. 

The  Army  should  also  ronsider  applying  maturation  development  to  any  fielded 
system  that  meets  ~  -  *  nteria: 

1.  It  is  scheduled  for  a  major  upgrade. 

2.  The  upgrade  involves  high-tedi  electronics  suites  (usuaUy  die  case  with 
major  upgrade  programs). 

3.  The  upgrade  will  apply  to  enough  of  die  fielded  fleet  (a)  to  permit  die 
collection  of  adequate  data  during  die  maturation  development  phase  and 
(b)  to  justify  the  cost  of  die  maturation  development  program. 
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To  the  degree  that  the  propoMd  electronic  suite  idatet  to  the  odftii^  suite,  ttie 
data  on  die  fielded  systems  may  be  used.  The  use  of  block  in^HOvements  and 
diorough  testing  using  "test  bed*  weapons  are  ootainly  consistent  widi  the 
maturation  development  concept 

Mote  specifically,  the  Army  could  test  the  maturatian  development  concept  on  a 
fielded  system  by  applying  it  to  the  upgrading  of  the  fire  control  system  of  die 
Apache.  Unlike  die  case  of  die  Air  Force  widi  die  F-15  and  F-16,  the  Army  may 
not  need  to  conduct  a  special  data  collection  effort  on  the  ^>ache  to  identify  and 
isolate  the  sources  of  RAM  performance  degradations.  The  contractor,  Martin 
Marrietta,  maintains  an  extensive  database  diat,  thou^  not  sufficient  to  support 
application  of  die  maturation  development  concept  contains  many  of  die  data 
above,  including  sane  operational  data  on  anonalies  at  the  end  item. 


Obstacles  to  Implementing  Maturation  Development 

Maturation  development  has  been  advocated  by  RAND  since  die  inid-198QB.  The 
logic  and  evidence  profiered  in  support  of  die  concept  have  been  widely 
accepted.  Why,  dien,  has  maturatum  development  not  been  implemented  in  any 
of  die  Services'  major  acquisition  or  upgrade  programs?  There  are  four  kinds  of 
obstacles  to  implementing  maturation  development;  cost  obstacles,  technical 
obstacles,  environmental  obstacles,  and  oganizational  obstacles. 

Cost  Obstacles 

One  obstacle  to  die  implementation  of  maturatian  development  is  that  program 
managers  are  reluctant  to  add  activities  (costs)  when  they  are  unsure  of  the 
savings;  moreover,  even  if  they  believe  that  savings  will  accrue  in  die  long  term, 
over  die  life  of  the  system,  they  may  be  unwilling  or  unable  to  implement 
additional  activities  that  increase  costs  in  die  short  term.  Costs  associated  with 
maturation  development  are  basically  of  two  sorts:  die  costs  of  developing, 
maintaining,  and  using  an  integrated  data  system  (including  die  costs  of  die 
intensive  operation  and  data  coUectioi  and  analysis  that  occurs  before  full 
fielding)  and  (2)  the  costs  of  developing  and  acquiring  new  component  designs 
that  ofier  improved  RAM  performance. 

To  our  knowledge,  no  analyses  to  date  have  calculated  the  savings  in  support 
costs  that  could  have  been  achieved  dirough  die  use  of  an  integrated  database  to 
improve  the  performance  of  the  maintenance  system.  Several  programs, 
however,  offer  evidence  regarding  dw  cost  savings  to  be  achieved  dirough 
component  redesign  to  improve  RAM  performance. 
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In  the  19605,  the  Air  Force  redesigned  the  missile  guidance  subsystem  for  die 
Minutmnan  I,  improving  its  MTBRsfrmn  600  hours  to  9000  hours.  This 
development  effort  cost  $150  million  but  saved  $15  billion  in  life-cycle  suppmt 
costs.  It  also  increased  the  availability  tevel  of  ttte  missile  from  70  percent  to  ova- 
95  percent.* 

The  Army's  Hawk  missile  system  included  a  successful  effort  to  improve  dre 
availability /resource  unit  cost  ratio.  The  Hawk  program  had  R&M  problems 
widv  the  high-power  illumirutor,  which  the  sample  data  collection  system 
indicated  had  a  mean-time-between-hulure  (MTBF)  rate  of  43  hours.  In  1979  the 
program  spent  $10.7  million  for  researdi  artd  develr^>ment  of  a  modified  unit. 

The  production  of  die  modified  units  cost  anodier  $164.5  million  in  19^  and  the 
training  and  installation  in  1985  cost  anodier  $21  million.  The  savings  in 
replenishment  spares  was  estiinated  at  $34.4  million  per  year.  Thecostsand 
dollar  benefits,  discounted  at  10  percent  per  year,  were  about  equal.  M<»e 
important  was  the  increased  availability  achieved  by  increasing  the  MTBF  from 
43  hours  to  134  hours  and  die  additional  benefit  of  a  range  improvement  of  20 
percent  Increases  in  availability  are  rarely  considered  in  diese  analyses  because 
of  the  difficulty  in  putting  a  dollar  valttt  on  them.* 

The  Air  Force  experience  in  applying  elements  of  the  maturation  development 
concept  to  the  F-15  and  F-16,  as  indicated  in  Section  2,  was  less  successful.  This 
experience  suggests  diat  if  die  component  design  problems  are  not  identified  early 
enough,  the  dollar  costs  to  incorporate  <»ily  die  maintainability  modifications 
generally  outweigh  die  costs  to  continue  to  support  deficient  weapon  systems 
logistically.  The  Air  Force  conducted  separate  studies  of  die  F-15  and  F-16  after 
they  were  fielded  to  obtain  die  data  needed  to  correctly  identify  die  problems 
discovered  during  (^leration.  bi  an  efibrt  to  reduce  the  costs  associated  with  die 
maintainability  problems,  die  program  managers  for  bodi  weapon  systems 
developed  ECPs.  In  the  case  of  the  F-16,  the  program  manager  sou^t  redesign  of 
the  LPRF  LRU,  and  in  die  case  of  die  F-15,  die  program  manager  sought 
replacement  of  the  BIT  system.  In  both  cases,  die  cost  to  develc^  new  systems  was 
about  $25  million  but  to  retrofit  die  existing  faulty  systems  fleetwide  would  cost 
another  $200  million.  As  a  result,  neither  modification  was  judged  to  be  justifiable 
on  the  basis  of  operating  and  support  costs  (despite  die  potential  improvement  to 
the  availability  of  FMC  systems),  and  the  modification  efforts  were  abandrmed. 

The  Army  can  learn  an  important  lesscm  from  the  Air  Force's  experience- 
namely,  that  it  is  imperative  to  begin  R&M  maturation  efforts  as  early  as 


*See  Gebnun  and  Siulman  (1968). 

*Based  on  unpublished  information  from  VS.  Army  Missile  Command  (17-19  July  1991  visit). 
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possible,  not  only  to  nuudmize  any  potential  suf^KMtt  costs  savings  over  the  Ule 
cycle,  but  to  be  able  to  realize  savings  suffickntly  large  enou^  to  oitset  the  costs 
a«gr»ria»«iri  widi  a  maturation  development  process  and  associated  engineering 
modificatkms. 

The  Air  Force's  inability  to  justify  these  modifications  raises  toe  osue  toat  rrew 
ways  of  measuring  the  costs  and  benefits  of  improving  high-tech  R6cM 
perfomumce  should  be  considered.  Simply  comparing  the  cost  of  a  modificaticm 
with  toe  cost  of  logistically  supporting  a  deficient  system  does  trot  capture  the 
cost  and  risks  of  living  wito  a  deficient  system,  in  terms  of  lost  weapon  system 
capability  or  lost  weapon  system  availability,  or  toe  benefits  of  truly  ntission 
capable  systems.  Further,  as  the  U5.  military  moves  to  keeping  weapon  systems 
Irmger  and  moves  to  a  CONUS-based  force,  toe  costs  of  logistically  supporting 
from  afar  need  to  be  strongly  considered.  Likewise,  toe  berrefit  of  bemg  able  to 
deploy  fewer  truly  mission  capable  weapon  systems  needs  to  be  captured  in  such 
an  arudysis. 


Technical  Obstacles 

The  chief  technical  obstacle  to  rruituration  development  is  the  rreed  to  adopt  a 
new  conceptual  framework  for  understanding  and  maiuiging  R&M  performarKe 
of  high-tech  systems-what  an  earlier  RAND  report  called  toe  need  for  "a  rtew 
view  of  weapon  system  reliability  arrd  maintainability."^  This  rrew  view  must 
focus  on  the  inherent  complexity  of  Type  B  faults  compared  wito  Type  A  faults 
and  toe  implications  for  data  collection,  attalysis,  and  oneasures  of  R&M 
perfomumce. 

The  difficulty  of  observing  Type  B  faults  stems  from  toeir  dependerue  on  a  larger 
set  of  ccmditions  in  operatioctal  time  sequerKes.  These  deperulendes  mean  that 
large  sample  sizes  are  needed  to  identify  atul  solve  these  fiiults.  Type  B  fiiults 
must  be  identified  throu^  toeir  recunerKe  in  a  sample  ratoer  toan  being 
identified  by  a  single  static  trutinterumce  dteck  as  are  Type  A  fiiults.  Witoout  a 
large  sample  size,  individual  anonudous  events  may  be  too  readily  ascribed  to 
"noise." 

Another  related  technical  obstacle  to  maturation  development  is  that  special 
testing  data  will  often  be  required  to  diagnose  the  cause  of  a  T3q>e  B  fault 
Indeed,  collecting  sudi  data  is  one  of  toe  purposes  of  toe  intense  period  of 
operation  data  collection  and  arudysis.  However,  all  data  toat  rruy  be  needed  on 


^^^Getmuoi  et  al.  (19W). 
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occasion  need  not  be  collected  all  ttie  time.  The  data  adlectkm  syston  can  be 
designed  to  peimit  specific  data  to  be  targeted  for  more  detailed  cdlection  after 
an  anomaly  has  been  observed. 

Lack  of  appropriate  measures  is  another  obstacle.  New  measures  of  R&M 
performance  are  ireeded  because  the  current  measures  were  devel<^)ed  at  a  time 
when  easily  r^>eated  Type  A  faults  domiiuited  the  RAM  problem.  These 
measures  do  not  attempt  to  define  aiul  m^ure  the  occurrence  of  Type  B 
problems.  For  example,  MTBF  is  a  good  measure  of  Type  A  problems  in  a  power 
supply  rectifier  when  the  fiulute  means  no  power,  but  it  is  not  useful  for  Type  B 
problems-for  instance,  degradaticKt  that  arises  because  in  some  operational 
modes  dte  power  supply  generates  substarrdard  output 

Configuraticm  control  is  also  a  tedmical  obstacle.  A  system  obviously  must 
undergo  design  changes  as  problems  are  urtcoveied  and  resolved  during  testing. 
Urdortunately,  the  effect  of  the  changes  on  complex  Type  B  faults  is  difficult  to 
evaluate  because  of  fire  necessarily  limited  testing  and  the  cumulative  or 
interactive  effect  of  the  changes.  As  dunges  are  introduced,  they  act  in 
combination  both  to  change  system  bdiavior  by  removing  and  introducing. 

These  combinations  introduce  new  configurations  tiuit  in  turn  reduce  the  sample 
size  per  configuration  in  the  data  collection. 

Given  tiie  developments  in  data  processing  and  tite  potential  for  onboard  data 
capture  in  operation,  tiie  technical  limitations  of  data  integration  and 
management  appear  much  less  serious  now  titan  even  a  few  years  ago. 
Unfortunately,  the  lack  of  measures  to  indicate  tiie  extent  and  cost  of  tiiese 
problems  in  fielded  systems  have  limited  tiie  appit^riate  application  of  such 
technology.  Lack  of  a  perceived  need  has  limited  the  automatic  collection  of  data 
from  databuses,  the  introduction  of  programmable  data  collection  using  such 
buses,  and  the  use  of  R&M  programmable  databuses. 

Environmental  Obstacles 

When  maturation  development  was  first  developed  in  the  1980s,  a  major  obstacle 
to  its  adoption  was  the  acquisition  environment  of  the  Cold  War  era.  When  the 
U.S.  faced  a  very  strcmg  Soviet  threat,  weapon  systems  were  designed, 
developed,  and  fielded  very  quickly  because  their  superior  capability  was 
needed  to  overmatch  tiie  latest  generation  of  Soviet  s3rstems.  Maturation 
development  was  resisted  because  it  would  require  additional  activities 
preceding  full-rate  production  and  tiiese  activities  could  delay  fielding  by  as 
much  as  a  few  years.  Altiiough  tiiese  activities  would  increase  true  weapon 
system  availability  of  fully  capable  systems,  improve  system  R&M  performance. 
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and  reduce  life-cycle  support  costs,  it  was  deemed  m<»e  piudait  to  field  weapon 
systems  that  were  not  fully  mature  in  terms  of  their  R&M  performaiu%.  The 
relative  R&M  immaturity  of  these  systems  drove  up  their  support  costs,  but  these 
cost  increases  were  accepted  as  part  of  die  price  of  security  in  the  Cold  War. 

Today  the  naticmal  security  environment  is  radically  different,  and  in  at  least  two 
respects  it  has  become  much  more  conducive  to  the  applicatum  of  maturation 
development.  First,  the  chief  motivation  for  highly  compressed  acquisition 
programs  has  disappeared  because  die  United  States  is  no  longer  in  an  intense 
arms  race  with  the  Soviet  Union.  Second,  because  acquisition  budgets  have  been 
greatly  reduced,  there  will  be  more  inclination  to  upgrade  an  existing  system 
than  to  acquire  a  new  one,  and  significant  upgrades  present  the  opportunity  to 
mature  the  R&M  performance  of  the  design  as  well  as  its  mission  capability. 
When  weapon  systems  can  undergo  a  more  deliberate  acquisition  process  and 
when  existing  systems  undergo  significant  upgrades,  time  to  mature  die  design 
is  available.  Third,  because  of  budgetary  pressures,  die  concern  widi  reducing 
total  life-cycle  support  costs  has  increased.  For  all  of  diese  reasons,  maturation 
development  should  appear  much  more  attractive  in  the  current  acquisition 
environment. 

On  the  other  hand,  some  features  of  die  new  environment  may  still  act  as 
obstacles  to  the  implementation  of  maturation  development  For  one  thing,  there 
are  likely  to  be  few  new  program  starts  and  thus  f^  opportunities  for  applying 
maturation  development  to  new  weapon  systems.  Second,  a  few  acquisitiim 
programs  may  still  be  hig^y  compressed—for  example,  if  they  are  developed  in 
response  to  a  new  and  unforeseen  technological  threat.  Third,  some  acquisition 
programs  may  involve  very  small  production  runs-periiaps  well  below  100 
weapon  systems— as  with  the  F-117  stealdi  filter  and  die  B-2  stealdi  bomber. 
When  such  small  fleets  of  weapon  s)r5tems  are  acquired,  it  may  not  be  possible  to 
justify  some  features  of  maturatim  development,  such  as  the  use  of  an  intense 
period  of  operation,  data  collection,  and  analirsis  before  full  fielding.  Sufficient 
data  may  only  be  generated  when  die  entire  fleet  is  operational.*^  The  most 
common  use  of  maturation  development  in  die  post-Cold  War  era  will  probably 
be  to  improve  the  R&M  performance  of  major  fielded  weapon  systems  scheduled 
for  a  significant  upgrade. 


A  detailed  discussion  of  the  utility  of  maturation  development  under  three  possible  acquistion 
regimes  in  the  post-Cold  War  era  appears  in  impublished  RAND  researdi  by  Frank  Camm  and 
Hyman  Shulmim. 
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Organizational  Obstacles 

A  nujor  source  of  obstacles  to  implementing  maturation  development  is  the  need 
for  diverse  organizations-both  <»gank  and  contractor-to  dumge  dwir  activities 
and  their  interactions. 

For  example,  one  organizational  obstacle  to  the  implementatum  of  maturatian 
development  is  tt\e  need  to  include  in  development  planning  dw  time  and 
resources  for  a  period  of  intensive  (^>eration,  data  coUecticm,  and  analysis. 
Changing  development  planning  in  this  way  requires  die  government  to 
acknowledge  that  there  will  be  R&M  problems  that  lie  outside  die  contractor's 
development  efforts.  This  conflicts  with  the  widespread  belief  that  the  prime 
contractor  is  responsible  for  ail  system  plwnomena  whedier  or  not  they  are 
included  in  the  system  testing.  This  bdief  persists  despite  die  absence  of 
examples  where  such  resproisibility  has  been  imposed  on  a  prime  contractor  (Le., 
cases  where  the  prime  has  been  held  responsible  for  complex  R&M  problems  that 
result  in  fleet  degradation  and  life-cycle  cost  increases,  as  opposed  to  cases  of  a 
properly  functioning  component  failing  to  perform  to  ^ledfication).  In  fact,  such 
general,  implied  responsibility  is  inappropriate  given  the  complex 
interdependency  of  respcmsibility  in  the  approval  and  execution  of  numerous 
design  changes  during  a  system  development  Sole  contractor  responsibility 
must  necessarily  be  limited  to  areas  where  there  is  a  clear  mutual  understanding 
of  the  responsibility  and  an  appropriate  means  of  pricing  that  responsibility. 

Another  organizational  obstacle  is  the  split  between  the  procurement  and 
support  organizations.  Maturation  development  requires  the  closer  integration 
of  these  functimial  stovepipes.  Currendy,  just  at  die  time  dut  a  large  sample  of 
data  from  fielded  systems  becomes  available,  the  major  responsibility  for  the 
support  and  cost  of  support  is  supposed  to  move  to  the  major  supporting 
command  and  depot  community.  The  procurement  prefect  office 
understandably  feels  that  it  has  completed  its  job,  and  the  depot  community  is 
occupied  with  assuming  engineering,  budgeting,  and  repair  responsibility.  The 
traditional  focus  of  program  managers  has  been  to  field  an  elective  weapon 
system  on  schedide  and  within  budget.  The  measure  of  "efiective"  has  usually 
been  the  ability  of  the  weapon  system  to  perform  its  missions.  And  die  budget 
concern  has  generally  been  limited  to  allocated  dollars  (current  year)  or  dollars  to 
be  allocated  to  conqilete  die  development  and  to  acquire  the  required  quantity. 
Future  costs  are,  arguably,  difficult  to  measure.  Maturation  development  is 
concerned  widi  providing  sustained  avaibbility  of  FMC  weapon  systems  and 
reducing  total  life-cycle  costs.  This  can  be  accomplished  when  acquirers  and 
logisticians  recognize  the  up-fiont  costs  and  the  life-cycle  benefits  of  a 
maturation  development  approach.  The  maturation  development  approadi 
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for  time  and  dollar  resources  to  mature  the  design,  the  requirement  to  manage 
R&M  risk,  and  "fault  removal  efficiency"  as  a  milestone  criterion. 

Underlying  organizational  difficulties  ate  die  complexity  of  die  Type  B  problems 
and  the  lack  of  appropriate  measures.  These  problons  require  a  team  approach 
that  is  very  difficult  to  achieve  in  an  organizadon  of  separate  functumal  groups 
with  separate  functional  responsibilities.  In  such  organizatimis,  integraticm 
across  die  functions  to  solve  a  complex  problem  does  not  take  place  when  it  is 
not  perceived  to  affect  die  performance  measures  by  which  die  groups  are 
evaluated.  For  example,  maintenance  technicians  currenUy  are  measured  on  the 
number  of  units  that  they  restore  to  operation.  Compmients  diat  show  NEOF  are 
ccmsidered  repaired  and  count  as  a  "restoradoi."  Althou^  nothing  was 
repaired  and  the  maintenance  activity  was  minimal,  maintenance  technicians 
receive  as  much  credit  as  though  they  consumed  10  times  the  resources  to  restore 
a  truly  broken  component.  Maturation  development  would  call  for  measures 
that  incentivize  true-fault  removal  effidency,  i.e.,  where  a  fault  is  only  considered 
removed  after  die  component  has  been  proven  to  function  properly  in  its 
operational  envirtmment.  If  an  NEOF  occurred  and  die  component  repeated  as  a 
failure,  maintenarKe  technicians  would  seek  to  identify  the  cause  of  the  NEOF. 


Conclusion 

This  report  has  demonstrated  the  following  points; 

1.  High-tech  systems  have  fault  detection  and  fault  isolation  problems  when 
first  fielded. 

Z  Lemons  develop  over  time. 

3.  These  problems  cause  an  excessive  burden  on  die  logistics  structure. 

4.  The  availability  of  truly  mission  capable  weapon  systems  is  probably 
overestimated  by  current  data  systems. 

5.  A  concept  for  "maturing"  the  R&M  performance  of  hi^-tech  weapon 
systems  is  needed. 

6.  Elements  of  such  a  craicept,  called  maturation  development,  have  been  tested 
by  the  Services  and  found  to  be  promising. 

Underlying  the  maturation  development  concept  ate  several  hypotheses  (1)  fault 

detection,  reporting,  isolaticm,  and  removal  can  be  improved  dirou^  the  use  of 

an  integrated  data  system;  (2)  component  design  problems  that  degrade  R&M 

performance  can  be  identified  with  sufficient  informatian;  and  (3)  component 
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designs  with  improved  R&M  performance,  if  proposed  early  enough  in  the 
acquisition  process,  warrant  the  dollar  cost  to  implement  them.  These 
hypiotheses  apply  both  to  the  acquisition  of  new  weapon  systems  and  the 
acquisition  of  major  upgrades  and  modifications. 

We  predict  that  the  need  for  maturation  development  will  increase.  The  need  to 
achieve  required  weapon  system  availabilities  at  lower  costs  is  evident 
throughout  the  Army.  Evidence  from  bodi  Army  and  Air  Force  hig^-tech 
weapon  systems  points  to  serious  diagnostic  and  repair  problems,  as  well  as  the 
existence  of  lemons.  All  evidence  suggests  tiiat  R&M  immaturity  is  directly 
correlated  with  technological  complexity.  Therefore,  the  Army  can  expect  to 
discover  increasing  R&M  problems  as  its  inventory  of  weapon  systems,  both 
through  upgrades  and  new  procurements,  becomes  increasingly  sophisticated 
technologically. 

Using  current  methods,  the  cost  of  keeping  d\e  same  high  levels  of  availability 
for  the  most  sophisticated  Army  weap>on  systems  that  we  have  enjoyed  in  the 
past  ]£  prohibitive.  While  many  approaches  toward  achieving  an  improved 
"availability  per  unit  resource  cost"  are  possible,  we  argue  in  dus  report  for 
improving  the  maintainability  of  the  weapon  systems.  Improvements  in 
maintainability  through  maturation  development  offer  die  twin  benefits  of 
increased  availability  of  fully  capable  weapon  systems  and  lower  total  life-cycle 
support  costs. 


Appendix 

Weapon  System  Sustainment  Management 
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This  appendix  summarizes  a  ccmcept  for  revolutionizing  the  Army  logistics 
system.  The  concept,  called  Weapcm  System  Sustainment  Management  (WSSM), 
has  been  developed  at  RAND  in  ccmjunction  witti  senior  Army  logistics 
leadership.  The  WSSM  concept  syndtesizes  dte  results  of  a  very  large  body  of 
logistics  research  ccmducted  by  RAND  over  several  decades  for  the  Services  and 
the  Office  of  die  Secretary  of  I>efense.  Maturation  development  is  an  approach 
for  impleinenting  one  of  three  strategies  in  support  of  WSSM.^ 

The  Need  to  Revolutionize  the  Anny  Logistics  System 

The  current  logistics  system  was  designed  to  support  a  massive  European  war. 
Widi  the  end  of  the  Cold  War,  the  U.S.  military  is  being  downsized  and  reshaped 
to  meet  die  requirements  of  a  new  era  in  which  military  power  will  need  to  be 
projected  from  the  CONUS  to  any  number  of  contingencies  around  the  world. 

To  meet  the  support  needs  or  die  Army  in  this  new  era,  the  Army  logistics 
system  must  become  leaner,  more  flexible,  and  more  responsive:  leaner  because 
defense  budgets  will  no  longer  enable  die  Army  to  maindun  a  massive  logistics 
s}rstem;  more  flexible  because  die  Army  must  prepare  for  a  wide  range  of 
potential  contingencies  radier  than  focus  on  a  major  Eur(^>ean  case;  and  more 
responsive  because  of  increased  uncertainty  regarding  die  nature  of  die  direat 
and  because  neidier  forward  positioning  nor  host  nation  support  can  be 
assumed.  Figure  A.1  suggests  schematically  how  die  future  Army  logistics 
system  will  differ  radically  from  the  current  massive  system. 

The  envisioned  changes  are  so  great  diat  one  mi^t  question  whedier  they  are 
even  feasible.  However,  diere  are  grounds  for  optimism. 


^For  a  fuller  discussion  of  WSSM,  see  ).  Dumond  et  aL,  Weapon  System  Sustamment  Management: 
A  Concept  for  Revdutkmidng  the  Army  Logistics  System,  Annotated  Bribing,  RAND,  DB-104-A, 
fordunming. 
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Figure  A.1 — ^The  Army  Logistics  System  Must  Become  Leaner,  More  Flexible, 

and  More  Responsive 

Radical  new  management  techniques  have  enabled  the  best  commercial  firms  to 
become  leaner  and  more  flexible.  On  many  measures-such  as  inventory,  defects 
per  unit,  timely  delivery,  production  lead  time-these  firms  have  achieved  order- 
of-magnitude  improvements  of  die  sort  diat  the  Army  logistics  system  must 
strive  for.  Many  have  ai^ed  that  a  new  management  paradigm  is  emerging. 
This  paradigm  is  marked  by  an  increased  focus  on  die  customer,  the 
establishment  of  measurable,  customer-related  goals;  die  "reengineering"  of 
processes  to  achieve  the  goals;  and  continuous  product  and  process 
improvement.  WSSM  applies  similar  management  strategies  to  improve  the 
Army  logistics  system,  as  shown  in  Figure  A2: 

•  Focus  the  entire  system  on  the  customer's  needs. 

•  Design  and  redesign  weapon  systems  to  be  more  supportable. 

•  Design  and  maiuige  processes  to  be  more  responsive  and  efficient. 

This  appendix  addresses  each  of  these  strategies  in  turn. 


Focus  the  Entire  System  on  the  Customer's  Needs 

Currently  managers  throughout  the  Army  logistics  system  rely  (»n  local  measures 
that  are  not  directly  linked  to  a  common  system  goal.  For  instance. 


— '  Focus  the  entire  system 


Figure  — Management  Concepts  Similar  to  Those  Used  by  the  Best  Commercial 

Hims  Can  Improve  the  Army  Logistics  System 

transportation  managers  may  use  a  measure  such  as  full  truck  load  to  assess  the 
performance  of  their  assets;  likewise,  repair  shop  managers  may  use  a  measure 
such  as  rate  of  labor  or  equipment  utilization.  These  measures  encourage 
efficient  use  of  resources  locally,  but  they  do  not  provide  any  indication  of 
whether  a  specific  management  action  improves  fixe  efficiency  and  eHectiveness 
of  the  logistics  system  as  a  whole.  Successful  commercial  firms  teach  ttxe 
importance  of  focusing  on  dte  customer.  The  customer  for  die  logistics  system  is 
the  operational  commander  \^o  needs  logistics  support.  More  specifically,  die 
operational  cotrunander  requires  suffidoit  weapon  systems  to  perform  die 
plarmed  mission.  The  responsibility  of  the  logistician  is  to  manage  the  inputs 
(persormel,  capital,  materiel,  information,  etc.)  and  die  processes  (distribution, 
repair,  etc.)  of  the  logistics  system  so  as  to  provide  the  sustained  weapon  system 
availability  that  the  operational  commander  needs. 

The  logistidan  faces  an  additional  challenge  in  attempting  to  provide  this  output 
at  a  time  when  logistics  resources  are  being  reduced.  RAND  is  analyzing  two 
fundamental  ways  to  compensate  for  reduced  resources:  (1)  reengineering 
logistics  processes  to  make  them  more  effident  and  more  elective  and 
(2)  making  better  use  of  information  in  support  of  those  reengineered  processes. 
Logistics  managers  require  die  capability  to  control  resources  effectively  and  die 
capability  to  assess  the  performance  of  die  system  so  that  die  use  of  resources  can 
be  adjusted  accordingly.  The  assessment  capability  permits  logistics  managers  to 
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understand  how  their  dedsioits  affect  the  goal.  A  decision  support  tool  for 
assessment  is  useful  to  managers  in  two  ways.  First,  it  permits  them  to  anticipate 
problems  in  meeting  the  goals  of  the  operati<mal  commaxuler.  For  example,  it 
might  indicate  that  at  a  certain  point  in  the  plaimed  mission  the  commander 
would  not  have  available  the  needed  number  of  weapon  systems.  Seccmd,  when 
problenns  have  been  identified,  the  same  tool  can  be  used  to  assess  how 
alternative  policies  affect  the  performance  of  die  system. 

RAND  has  developed  assessment  tools  for  the  Army  and  the  Air  Force.  It 
developed  an  assessment  tool  called  Dyna-METRIC  for  the  Air  Force,  which  has 
implemented  it  as  part  of  its  Weapon  System  Management  Infonruition  System. 
RAND  recently  adapted  the  tool  to  the  needs  of  the  Army.  The  Army  has  field- 
tested  a  prototype  of  this  version  at  the  U.S.  Tank-Automotive  Corrunand  to 
assess  support  of  the  MlAl  Abrams  Tank  and  the  Bradley  Fighting  Vehicle.  The 
Army  and  RAND  have  further  experimented  with  using  the  tool  to  assess  die 
support  of  systems  in  Somalia. 


Design  and  Redesign  Weapon  Systems  to  Be  More 
Supportable 

Weapon  systems  create  die  burden  on  the  logistics  S3^tem.  U.S.  weaprwi  systems 
are  increasingly  complex  as  more  high-tech  (largely  digital)  components  are 
added  to  increase  capability.  Unfortunately,  this  added  complexity  in  weapon 
systems  also  results  in  reduced  availability  and  increased  costs.  As  Figure  A.3 
illustrates,  the  added  burden  comes  about  in  two  ways.  First,  high-tech 
components  and  subcomponents  do  not  usually  fail  outright  but  rather  exhibit 
spotty  suid  degraded  performance.  Such  failure  modes  are  hard  to  diagnose  and 
isolate,  with  the  result  that  maintainers  often  will  remove,  test,  and  repair  several 
components— most  of  them  good— in  search  of  the  foulty  component.  Second,  a 
few  individual  components  are  lemons— that  is,  they  are  chronically  defective 
and  cycle  through  the  repair  system  repeatedly.  These  lemons  account  for  about 
half  of  the  workload  on  subcomponents  at  their  respective  depot-level  repair 
shops.  Compared  to  a  non-lemon  component  of  die  same  design,  a  lemon 
consumes  20  times  as  many  subcomponents.  Both  the  fault  isolation  problems 
and  the  presence  of  lemons  in  weapon  systems  cause  commanders  to 
overestimate  the  number  of  available  systems  that  are  truly  FMC. 

During  the  Cold  War,  the  United  States  had  a  strong  rationale  for  fielding  new 
weapon  systems  that  were  not  fully  matured  in  terms  of  their  R&M.  Iheir 
operational  capability  was  needed  to  maintain  a  margin  of  technological 
superiority  over  the  Soviets.  This  rationale  has  diminished  in  the  post-Cold  War 
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Figuie  A3— More  Complexity  in  Weapem  Systems  Results  in  Reduced  AvsiUbilily 

end  Inocssed  Costs 

era,  and  the  United  States  can  now  adopt  a  less  compressed  acquisition  strategy 
that  will  permit  complex  weapon  systems  to  be  more  fully  matured  before 
fielding,  thus  reducing  their  burden  on  d\e  logistics  system. 

RAND  has  developed  a  concept  for  improving  dte  sustainability  of  weapon 
systems  in  order  to  achieve  increased  weapon  system  availability  at  lower  costs. 
This  concept  calls  for  maturing  dre  design  of  newly  developed  weapon  systems, 
particularly  during  the  low-rate  production  phase,  aiul  identifying  ttre  lemons 
during  die  fielded  phase.  Ihe  key  element  of  die  approadi  is  an  intensive  data 
collection  and  analysis  prior  to  full-rate  production.  The  approach  rails  for  die 
design  of  the  weapon  system  to  be  frozen  during  low-rate  production  so  that  a 
known  design  configuration  can  be  operated  intensely  while  data  is  collected  and 
analyzed.  Based  upon  the  results  of  diis  analysis,  the  Army  would  modify  the 
design  of  the  weapon  system  to  make  it  more  mature  and  supportable.  These 
improvements  would  lead  to  improved  availability  at  lower  costs  dirouj^out  die 
fielded  life  of  the  system.  To  support  die  intensive  analysis,  a  database  would  be 
established  during  the  earliest  phases  of  acquisition.  This  database  would  be 
integrated  across  time,  echelons,  and  functions  and  would  be  sustained  diroi^ 
the  life  of  the  s}^tem  in  order  to  identify  additional  design  problems  that  may 
emerge  due  to  ag^g  effects  or  to  mi^on  changes. 
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The  database  would  also  be  used  to  identify  and  remove  lemons.  These  are  h^- 
tech  components  that  exhibit  chronic  performance  problems.  Although  sud\ 
c<Mnponents  compose  about  9  percent  of  the  total  set,  they  are  re^nsible  for  a 
very  large  prc^>ortion  of  the  logistics  burden  and  contribute  very  little  to 
operational  capability.  These  lemons  can  be  identified  through  the  use  of  a 
database  that  tracks  components'  (^)eratiorud  aiul  maintenaiKe  history  by  serial 
number. 

RAND  has  recommended  ttut  die  Army  apply  this  concept  to  the  Comanche  and 
to  the  upgrade  of  die  Apache.  The  recommendadon  for  die  Comanche  emerged 
during  a  1987  study  of  the  Army's  proposed  new  attack  helicopter  (then  called 
the  Light  Helicopter  Experimental).  That  study  projected  diat  just  eight  high- 
tech  compcments  in  the  avionics  suite  would  account  for  80  percent  of  die  spares 
and  repair  costs  and  70  percent  of  the  system  downtime.  Maturation  of  these 
components  was  recommended  to  enable  the  program  to  achieve  its  performance 
and  cost  goals.  Serial  number  tracking  would  also  permit  the  culling  of  lemons. 

Although  the  Comanche  acquisition  is  delayed,  die  Army  can  gain  experience 
with  these  concepts  by  applying  them  to  the  planned  upgrade  of  the  Apache. 

The  concept  is  also  applicable  to  fielded  systems;  however,  because  the 
application  of  matu.'adon  development  affects  component  design,  some  of  its 
potential  benefits  will  be  offset  by  the  cost  of  retrofitting  an  existing  fleet. 

Design  and  Manage  Processes  to  Be  More  Responsive 
and  Efficient 

As  depicted  in  the  top  panel  of  Figure  A.1,  die  current  logistics  system  is  too 
costly,  slow,  and  inaccurate.  As  of  die  end  of  1992,  the  DoD  has  over  $80  billion 
in  spare  parts.  Yet  widi  all  this  mass,  die  system  still  is  not  responsive.  In  a 
study  of  support  in  Operation  Desert  Storm,  RAND  teseardiers  interviewed 
unit-level  commanders  and  logisticians  who  for  months  received  no  q>are  parts 
to  bring  out-of-commission  weapon  systems  to  missian-ready  status-even 
though  the  supply  system  shqiped  massive  stocks  to  the  dieater,  including 
twenty-five  diousand  (25,000!)  forty-foot  containers  whose  contents  were 
unknown.  The  system  certainly  did  what  it  was  designed  to  do-project  a 
massive  amount  of  materiel  forward-but  having  mass  does  not  necessarily 
provide  the  weapon  system  availability  needed  by  commanders. 

RAND  is  advocating  a  management  concept  called  "velocity  management"  that 
aims  to  replace  mudi  of  die  current  reliance  on  logistics  mass  to  a  reliance  on  the 
improved  velocity,  accuracy,  and  rdiability  of  logistics  processes.  The 
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commercial  sector  has  demcmstrated  that  die  speed,  accuracy,  and  reliability  of 
processes  can  be  dranuiticaUy  improved.  RAND  is  analyzing  ways  in  whidi 
Army-and  DoD  generally-logistics  processes  can  be  reengineered  to  achieve  the 
S2une  type  and  magnitude  of  improvements.  The  key  is  to  remove  non-value- 
adding  activities  and  to  improve  the  performance  of  value-adding  activities. 

With  RAND's  assistance,  die  Air  Force  tecendy  conducted  a  field  test  that 
demonstrated  how  reengineered  processes  can  lead  to  radical  improvements. 

The  Air  Force  "reengineered"  the  depot  repair  process  for  32  high-value  and  very 
high-value  components  in  400  aircraft.  The  result  was  a  75  percent  improvement 
in  turnaround  time  for  the  hig^-value  components  (from  32  to  8  diys)  and  an  81 
percent  improvement  for  very  high-value  ccnnponents  (horn  32  to  6  days).  The 
reengineered  s)rstem  saved  millions  of  dollars  per  year  and  delivered  the  same 
performance  as  the  old  system. 

Improved  velocity  of  logistics  processes  reduces  the  need  for  expensive 
inventory.  RAND  anal)rsis  of  Martin  Marietta  data  associated  widi  just  one  high- 
tech  component  of  the  Apache  helicopter  provides  an  example.  The  analysis 
showed  that  if  the  Army  could  increase  the  velocity  of  this  component  through 
the  depot  repair  pipeline  from  about  90  to  about  15  days,  dien  it  could  reduce  die 
value  of  stock  in  the  pipeline  from  about  $60  million  to  about  $10  million  (an  83 
percent  reduction). 

These  two  examples  from  the  Air  Force  and  die  Army  illustrate  process 
improvements  on  the  same  scale  as  those  found  in  the  best  commercial  firms. 
Logistics  managers  will  need  to  use  gieady  reduced  resources  more  effidendy 
and  effectively.  No  longer  will  it  be  possible  to  rely  on  massive  resources  to 
cover  uncertainty  and  risk.  Logistics  managers  wiU  need  decision  support  tools 
to  help  them  control  reduced  resources.  Such  tools  will  help  a  manager  decide 
how  to  use  available  repair  and  supply  resources  so  as  to  meet  the  needs  of  the 
operational  commander  in  the  most  efficient  manner.  RAND  has  developed 
control  tools  for  die  Army  and  die  Air  F<»ce.  The  RAND-devdoped  control 
methodology,  called  DRIVE  for  the  Air  Force,  is  undergoing  field  testing  at 
Ogden  and  San  Antonio  Air  Logistics  Centers.  RAND  also  adapted  die 
methodology  to  the  Army's  needs.  The  Army,  which  calls  the  system  RBM  or 
Readiness-Based  Maintenance,  has  field-tested  die  tool  at  the  U3.  Army  Missile 
Command  to  control  repair  of  the  Multiple  Launcher  Rocket  System.  RAND  has 
also  developed  another  version  of  die  tool  that  reflects  the  new  DoD  policy  to 
increase  die  operational  unit's  incentives  to  reduce  repair  costs. 
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Conclusion 

The  managemait  ccsKept  described  here.  Weapon  System  Sustainment 
Maiuigement,  integrates  much  RAND  logistics  research.  As  we  have  iiulicated, 
RAND's  experience  widi  assessment  tools  (the  Dyna-METRIC  family)  began 
with  the  Air  Force  and  grew  to  include  die  Army  as  welL 

Similarly,  the  maturation  development  cotKept  has  been  develc^ped  thrcHig^  a 
series  of  studies  addressing  the  rteeds  of  different  Air  Force  and  Army  systems. 
RAND'S  experieiure  with  the  DRIVE  family  of  ctmtrol  tools  also  includes  both  Air 
Force  and  Army  applications.  Some  of  dte  examples  of  improved  processes  aiul 
streamlined  logistics  structures  were  devel(^>ed  in  odier  RAND  projects  on 
modular  logistics,  alternative  support  structures,  and  alternative  maintenance 
concepts.  A  current  study  on  die  Army  distribution  system  is  also  contributing 
to  the  Weapcm  System  Sustairunent  Concept  RAND's  Weapxm  System 
Sustaiiunent  Concept  is  influencing  DoD  as  well  as  die  Army.  'The  DoD  is 
undertaking  a  thorough  review  of  die  existing  logistics  business  practices  in  die 
Services  with  the  goal  of  identifying  improved  processes  that,  to  die  extent 
possible,  are  standardized  across  Services.  Then  die  DoD  will  mandate  die 
development  and  implementation  of  standardized  logistics  information  systems 
to  support  those  improved  processes.  RAND  has  several  projects  ongoing  for 
clients  who  are  engaged  in  diis  activity  (including  the  Army,  Air  Force,  Navy, 
Defense  Logistics  Agency,  Assistant  Secretary  of  Defense  for  Production  and 
Logistics,  and  die  Director  of  Defense  Information)  and  so  is  well  positioned  to 
contribute  to  its  outcome.  We  believe  that  die  Weapon  System  Sustainment 
Management  concept  can  be  applied  to  achieve  die  goal  of  a  leaner,  more  agile 
logistics  system  and  that  RAND-developed  tools  such  as  Dyna-METRIC  and 
DRIVE  may  provide  the  basis  fen:  part  of  a  standardized  logistics  management 
system. 
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